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Abstract 


Department  of  Defense  (DoD)  spending  on  services  has  been  trending 
upwards  for  over  a  decade  and,  as  of  2011,  accounts  for  56%  of  total  contract 
spending.  The  increased  reliance  on  services  contractors  has  prompted  the 
Government  Accountability  Office  (GAO)  to  look  more  closely  at  the  acquisition  and 
contract  management  process.  The  term  program  management  describes  the 
approach  and  methodology  needed  for  the  management  of  complicated  projects. 

We  used  the  program  management  approach  to  address  the  following  questions: 

(1)  How  do  different  stakeholders  define  successful  services  contracts  within  the 
Navy?  (2)  How  do  different  stakeholders  measure  services  contracts  within  the 
Navy?  and  (3)  How  should  Navy  services  contracts  be  defined  and  measured?  We 
conducted  a  survey  of  168  key  stakeholders.  We  discovered  that  when  defining  and 
measuring  the  success  of  a  service  contract  all  stakeholders  tend  to  utilize  outcome- 
related  factors  over  process-oriented  factors.  We  believe  this  is  because  outcomes 
tend  to  drive  perceptions  of  success  more  than  processes  and  are  more  easily 
quantifiable.  Metrics  used  to  measure  success  are  typically  related  to  cost, 
schedule,  and  performance.  Based  on  these  findings,  we  provide  recommendations 
on  establishing  better  internal  control  measures,  putting  in  place  an  operational  audit 
process,  and  creating  a  standardized  reporting  process. 

Keywords:  Services  Contracts,  Services  Acquisition,  Stakeholder  Theory, 
Agency  Theory,  Program  Management,  Critical  Success  Factors  (CSFs) 
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I. 


Introduction 


A.  Background 

Department  of  Defense  (DoD)  spending  on  services  has  been  trending 
upwards  for  over  a  decade  and,  as  of  201 1,  accounted  for  56%  of  total  contract 
spending,  compared  to  48%  in  2000  (Berteau,  Ben-Ari,  Sanders,  Morrow,  &  Ellman, 
2012).  Figure  1  shows  DoD  spending  trends  across  six  broad  services  categories 
by  compounded  annual  growth  rate  (CAGR).  The  categories  used  are  information 
and  communications  technology  (ICT);  professional,  administrative,  and 
management  services  (PAMS);  research  and  development  (R&D);  equipment- 
related  services  (ERS);  facility-related  services  and  construction  services  (FRS&C); 
and  medical  services  (Berteau  et  al.,  2012).  The  Center  for  Strategic  and 
International  Studies  modified  the  standard  DoD  portfolio  grouping  by  adjusting 
services  codes  to  more  appropriately  align  with  their  respective  categories  (Berteau 
et  al.,  2012).  In  fiscal  year  (FY)  201 1,  more  than  half  of  the  DoD’s  $375  billion  in 
contract  obligations  was  for  services  (Government  Accountability  Office  [GAO], 
2012a).  The  DoD  has  steadily  increased  its  reliance  on  services  contractors  to 
augment  its  critical  capability  shortfalls  across  the  six  services  categories  shown  in 
Figure  1  (GAO,  2011b). 
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Service  Area 
(11-year  CAGR) 


2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011 


B  MED  (15.6%  11-year  CAGR) 

□  PAMS  (8.9%  11-year  CAGR) 

a  ICT  (6.1%  11-year  CAGR) 

Q  FRS&C  (5.9%  11-year  CAGR) 

■  ERS  (8.2%  11-year  CAGR) 
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Figure  1.  Defense  Contract  Spending  by  Services  Area  2000-2011 

(Berteau  et  al.,  2012) 

As  DoD  spending  on  services  has  increased  over  the  last  10  years,  numerous 
published  Government  Accountability  Office  (GAO)  and  Department  of  Defense 
Inspector  General  (DoDIG)  reports  have  cited  deficiencies  in  multiple  areas  of  the 
contracting  process.  Between  2001  and  2009,  the  GAO  issued  16  reports  citing 
deficiencies  in  the  acquisition  of  services.  Between  FY2003  and  FY2008,  the 
DoDIG  issued  142  reports  citing  weaknesses  in  the  acquisition  and  contract 
administration  process  (DoDIG,  2009).  Contract  management  has  been  on  the  GAO 
high-risk  list  since  1992,  revealing  the  difficulties  in  meeting  services  procurement, 
cost,  schedule,  and  performance  objectives  (Apte,  Apte,  &  Rendon,  2010).  Deficient 
process  areas  found  by  both  the  GAO  and  DoDIG  included  market  reasearch, 
contract  type,  requirements  management,  project  management,  contractor 
oversight,  and  personnel  training  (Apte  et  al.,  2010).  These  cited  problems  are 
compounded  by  an  acquisition  workforce  that  has  remained  the  same  size  since 
2001 ,  while  spending  services  have  doubled  over  the  same  period  (GAO,  2009b). 
The  DoD’s  contract  management  process  capabilities  have  also  been  found  to  be 
lacking  in  all  phases  of  the  contracting  process.  The  contract  administation  and  the 
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contract  closeout  phases,  specifically,  have  even  lower  process  capability  than  the 
other  phases. 

The  DoD  needs  to  focus  on  improving  services  contract  mangement  by  first 
identifying  how  disparate  stakeholders  define  and  measure  the  success  of  services 
acquisitions.  Research  is  necessary  to  gain  an  increased  understanding  of  differing 
stakeholders’  goals  and  objectives,  which  could  be  used  to  develop  a  standardized 
definition  of  services  acquisition  success  and  to  align  stakeholders  towards  a 
common  goal.  In  order  to  maximize  the  use  of  scarce  acquisition  resources  and 
improve  outcomes,  research  is  necessary  to  identify  and  provide  recommendations 
on  the  factors  of  successful  services  contracts. 

B.  Purpose 

The  primary  purpose  of  this  study  was  to  conduct  further  analysis  of  services 
contracting  management  practices  within  the  Navy.  The  objective  of  this  research 
was  to  build  upon  the  understanding  developed  in  prior  research  projects  to 
generate  metrics  for  defining  and  measuring  successful  services  contracts.  These 
developed  metrics  will  later  help  identify  factors  that  influence  successful  services 
contract  outcomes.  We  designed  a  survey  based  on  the  exploratory  findings  of 
Frank  Miller,  James  Newton,  and  Salvatore  D’Amato  (2012)  in  Defining  and 
Measuring  Success  of  Service  Contracts.  We  distributed  our  survey  to  the  primary 
stakeholders  for  services  acquisitions  to  determine  how  different  stakeholders  define 
and  measure  success.  The  findings  of  this  project  will  support  the  continued 
research  of  Dr.  Rene  G.  Rendon,  PhD,  and  Dr.  Uday  M.  Apte,  PhD,  on  the  DoD’s 
management  of  services  contracts. 

C.  Research  Questions 

In  our  efforts  to  build  upon  prior  research,  we  attempted  to  answer  the  same 
questions  that  Miller,  Newton,  and  D’Amato  presented  in  their  2012  project.  The 
three  research  questions  that  relate  to  services  acquisitions  are  as  follows: 
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■  How  do  different  stakeholders  define  successful  services  contracts 
within  the  Navy? 

■  How  do  different  stakeholders  measure  services  contracts  within  the 
Navy? 

■  How  should  Navy  services  contracts  be  defined  and  measured? 

D.  Benefits  and  Limitations  of  This  Project 

This  research  builds  on  the  exploratory  findings  from  the  2012  Miller,  Newton, 
and  D’Amato  project.  The  results  of  this  project  can  be  used  to  strengthen 
contracting  professionals’  understanding  of  successful  services  acquisitions  from 
disparate  stakeholder  perspectives.  Improved  understanding  of  this  subject  will 
enable  the  development  of  metrics  to  measure  the  success  of  services  contracts.  In 
this  study,  we  attempt  to  identify  the  cogent  elements  that  drive  increased 
performance  and  decreased  cost  for  taxpayers. 

This  research  was  limited  by  the  sample  size  of  stakeholders  surveyed 
relative  to  the  population  of  stakeholders  involved  in  services  acquisitions.  Due  to 
limited  availability  of  stakeholder  subjects,  obtaining  a  larger  sample  size  was  not 
feasible  for  this  project. 

E.  Research  Methodology 

We  addressed  the  research  questions  with  an  anonymous  online  survey  sent 
to  stakeholders  employed  by  the  Department  of  the  Navy.  The  survey  contained 
seven  demographic  questions,  three  open-ended  questions,  seven  multiple-choice 
questions,  and  four  ranking  questions.  This  survey  was  adapted  from  the  original 
survey  developed  by  Miller,  Newton,  and  D’Amato  (2012).  We  used  the  findings 
from  their  research  to  refine  questions  and  limit  potential  ambiguity  in  data  collection. 

F.  Organization  of  Report 

This  report  is  organized  into  five  chapters.  Chapter  I  includes  background 
information,  the  purpose  of  the  report,  research  questions,  benefits  and  limitations  of 
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the  project,  and  the  research  methodology.  In  Chapter  II,  we  review  literature  on  the 
services  contracting  process,  Agency  and  Stakeholder  Theories,  the  various 
stakeholders  involved,  and  critical  success  factors.  We  examine  various  GAO  and 
DoDIG  reports  and  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  (USD[AT&Lj)  guidance  concerning  current  weaknesses  in  the  services 
contracting  process.  In  Chapter  III,  we  explain  how  we  collected  and  analyzed  our 
data  and,  specifically,  how  we  created  our  survey  instrument  and  our  analytical 
process.  We  also  provide  a  brief  description  of  the  commands  that  participated  in 
the  survey.  In  Chapter  IV,  we  provide  our  survey  findings  and  discuss  our  research 
results.  In  Chapter  V,  we  provide  a  summary  of  and  conclusions  related  to  our 
research,  and  we  also  offer  recommendations  for  improving  services  acquisitions  in 
the  Navy.  We  also  provide  areas  for  further  study  in  DoD  services  acquisitions. 

G.  Summary 

In  Chapter  I,  we  provided  background  information  on  services  acquisitions, 
the  purpose  of  the  research,  research  questions,  benefits  and  limitations  of  the 
research,  and  the  methodology  and  organization  of  the  report.  Spending  on 
services  contracts  has  doubled  over  the  last  10  years,  despite  an  acquisition 
workforce  that  has  not  expanded  to  meet  this  increased  workload.  Indeed,  the  DoD 
has  significant  challenges  to  address  in  contract  management  to  remove  the  GAO’s 
high-risk  billing.  The  primary  focus  of  this  report  is  the  research  questions 
presented.  In  Chapter  II,  we  review  literature  on  deficiencies  in  services  contracting, 
a  program  management  approach  to  services  acquisitions,  the  services  contracting 
process,  Agency  and  Stakeholder  Theories,  stakeholders  involved  in  the  process, 
and  critical  success  factors. 
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Literature  Review 


A.  Introduction 

In  Chapter  I,  we  established  the  foundation  of  this  research.  In  this  chapter, 
we  provide  a  literature  review  of  services  contracting.  This  review  covers  the 
following  topics:  deficiencies  in  services  contracting,  the  need  for  a  program 
management  approach  to  the  services  contracting  process,  the  current  services 
contracting  process,  Stakeholder  Theory  and  the  various  stakeholders  on  the 
acquisition  team,  critical  success  factors,  and  the  application  of  Agency  Theory  to 
the  services  contracting  process.  We  also  show  how  both  Agency  Theory  and 
Stakeholder  Theory  can  be  tied  to  contracting  for  services  and  describe  why  there  is 
a  need  for  a  program  management  approach  to  the  process.  In  the  next  section,  we 
examine  specific  discrepancies  that  are  noted  in  GAO,  USD(AT&L),  and  DoDIG 
reports. 

B.  Deficiencies  in  Services  Contracts 

DoD  services  contract  obligations  have  risen  from  $92  billion  in  2001  to  over 
$200  billion  in  2008  (GAO,  2009a).  In  FY2006,  more  funds  were  obligated  for 
services  contracts  than  for  supplies  and  equipment  combined  (GAO,  2007a).  This 
massive  spending  growth  in  services  is  not  attributable  solely  to  the  sizable  logistical 
support  efforts  required  in  Iraq  and  Afghanistan.  In  2008,  these  operations 
accounted  for  $25  billion  of  the  $200  billion  spent  on  services,  but  the  remaining 
growth  is  due  to  other  factors  (GAO,  2009a).  These  factors  include  contractors 
filling  roles  previously  held  by  government  employees  through  outsourcing  and  the 
DoD  using  services  contracts  for  historically  non-services  acquisitions  (GAO, 

2007a).  It  is  important  to  note  that  during  the  recent  years  of  defense  spending 
drawdown  (2008-2011),  spending  on  services  has  decreased  at  a  lower  percentage 
rate  than  spending  on  products  (Berteau  et  al.,  2012). 
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The  significant  growth  in  services  spending  discussed  previously  is 
accompanied  by  a  contracting  workforce  that  has  remained  relatively  the  same  size 
since  2001  (GAO,  2009b).  Prior  to  2001,  the  acquisition  workforce  was  slashed  by 
nearly  50%  from  1989  to  1999  (GAO,  2012c).  A  sufficiently  trained  and  competent 
acquisition  workforce  is  necessary  for  effective  contract  management  and  is  needed 
to  achieve  successful  defense  acquisition  outcomes  (Apte  et  al.,  2010).  The  GAO 
(2009b)  reports  that  acquisition  workforce  capability  shortfalls  make  it  difficult  to 
ensure  that  value  is  achieved  and  expose  the  Department  of  Defense  to 
unnecessary  risk.  Figure  2  shows  the  shrinking  acquisition  workforce  as 
procurement  appropriations  soared  since  1998. 
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Figure  2.  DoD  Procurement  Appropriations  and  Acquisition  Workforce 

(Gansler,  2011) 

In  this  section,  we  look  further  into  the  deficient  contracting  areas  noted  by 
the  DoDIG  and  GAO.  Government  reports  indicate  insufficient  or  undocumented 
market  research  during  the  acquisition  planning  process  for  services  (DoDIG,  2009; 
GAO,  2012b).  Market  research  is  required  by  the  Federal  Acquisition  Regulation 
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(FAR,  2012),  and  its  purpose  is  to  reveal  the  market’s  capability  of  meeting  the 
government’s  specific  acquisition  requirement  and  to  assist  in  determining  the 
appropriate  acquisition  strategy  to  fulfill  it  (FAR,  2012).  Market  research  is  also 
necessary  to  enhance  competition  and  to  ensure  government  socioeconomic 
objectives  are  achieved.  A  primary  government  socioeconomic  objective  is  to 
promote  opportunities  for  small  business  concerns  to  compete  for  government 
acquisitions.  The  government  uses  small  business  set-asides  to  award  certain 
acquisitions  exclusively  to  small  business  concerns  and  more  specific  categories 
such  as  services-disabled  veteran-owned  small  businesses  (FAR,  2012).  Market 
research  will  determine  if  an  acquisition  is  suitable  for  set-aside  by  determining  if 
responsible  small  business  concerns  can  satisfy  the  government’s  requirement. 

The  GAO  and  DoDIG  have  found  inappropriate  contract  types  used  on 
services  contracts,  which  leads  the  government  to  shoulder  increased  cost  risk 
(DoDIG,  2009;  GAO,  2009a).  Specifically,  the  GAO  found  the  overuse  of  high-risk 
time-and-materials  contracts  due  to  their  ease  of  use,  speed,  and  flexibility  (GAO, 
2009a).  The  use  of  time-and-materials  contracts  should  be  limited  because  they 
provide  no  incentive  to  the  contractor  for  cost  control  and  efficiency  (GAO,  2009a). 
There  is  not  a  blanket  good  or  bad  type  of  contract,  only  the  inappropriate  use  of  a 
specific  contract  type  in  a  procurement.  The  appropriate  contract  type  is  necessary 
to  properly  distribute  contract  risk  between  the  government  and  contractor,  and  to 
provide  the  contractor  with  the  most  incentive  for  efficient  performance. 

The  use  of  a  proven  program  management  approach  is  considered  a  best 
practice  for  managing  services  contracts  (Apte  et  al.,  2010).  A  program 
management  approach  utilizes  methods  such  as  formal  project  managers,  project 
teams,  integrated  processes,  and  a  project  life  cycle  to  manage  contracts.  Despite 
these  proven  best  practices,  government  reports  have  shown  that  the  DoD  does  not 
have  an  adequate  management  structure  to  oversee  services  acquisitions  at  both 
the  strategic  and  transactional  levels  (DoDIG,  2009;  GAO,  2009a). 
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Government  reports  have  repeatedly  identified  contractor  oversight  and 
contract  administration  as  serious  problems  in  services  contract  management 
(DoDIG,  2009;  GAO,  2007b,  2007c,  2009a).  Proper  oversight  and  administration 
ensure  that  contractors  are  providing  timely  and  quality  services  in  accordance  with 
the  terms  of  the  contract  and  mitigate  contractor  performance  risk  throughout  the 
period  of  performance. 

Both  the  GAO  and  DoDIG  have  identified  poorly  defined  services 
requirements  as  a  deficient  area  of  services  contracts  (DoDIG,  2009;  GAO,  2009a). 
A  clearly  defined  description  of  what  the  contractor  is  required  to  provide  the 
government  is  necessary  to  effectively  hold  contractors  accountable,  meet  customer 
needs,  and  ensure  that  the  best  value  is  achieved. 

The  noted  deficiencies  discussed  in  this  section  provide  insight  into  why 
contract  management  has  been  on  the  GAO  high-risk  list  for  20  years. 

Improvements  in  multiple  contract  management  areas  are  required  to  effectively 
reduce  the  government’s  exposure  to  the  risk  of  overpaying  for  services. 

With  DoD  financial  resources  declining  since  2009,  the  DoD  must  achieve 
optimal  value  for  defense  acquisitions.  In  his  2010  Better  Buying  Power  Guidance 
Memorandum  (OUSD[AT&L]),  the  Under  Secretary  of  Defense  (Acquisition, 
Technology,  and  Logistics;  USD[AT&L])  acknowledged  that  DoD  practices  for 
services  procurement  are  far  less  mature  than  for  weapons  systems,  and  he 
provided  guidance  to  improve  services  acquisition  efficiency.  The  USD(AT&L) 
directed  each  branch  to  appoint  a  flag-level  senior  manager  for  services,  adopt  a 
standard  taxonomy  for  types  of  services,  and  address  the  root  causes  of  poor 
tradecraft  in  services  (OUSD[AT&L],  2010).  The  component  senior  manager  for 
services  will  be  responsible  for  governing  the  planning,  execution,  strategic  sourcing, 
and  management  of  services  contracts  (OUSD[AT&L],  2010).  This  USD(AT&L) 
initiative  takes  steps  to  address  weaknesses  cited  by  the  GAO  in  strategic 
management  of  services  acquisitions.  A  standard  taxonomy  of  reporting  categories 
will  ensure  consistency  across  and  within  the  departments  and  improve  visibility  and 
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the  ability  to  measure  productivity  and  success  across  the  DoD.  The  USD(AT&L) 
cited  the  following  examples  of  poor  tradecraft  that  must  be  improved:  mission 
creep,  one-bid  competitive  procurements,  misuse  of  time-and-materials  and  award 
fee  contracts,  and  the  need  to  incentivize  productivity  for  large  services  contracts 
(OUSD[AT&L],  2010).  The  Secretary’s  strategic  guidance  is  an  important  step  in 
addressing  the  problems  found  by  the  GAO  and  DoDIG,  and  it  provides  a  path  for 
delivering  better  value  to  the  taxpayers. 

The  deficiencies  described  previously  are  significant  and  by  no  means 
comprehensive.  There  are  numerous  other  examples  of  insufficient  oversight,  lack 
of  knowledge  by  acquisition  personnel,  and  improper  contract  administration.  Some 
of  the  deficiencies  listed  could  be  corrected  if  a  program  management  approach  was 
widely  implemented  by  the  DoD  into  the  services  contracting  process.  In  the  next 
section,  we  give  a  brief  overview  of  program  management  concepts  and  how  they 
could  be  applied  to  services  acquisitions. 

C.  Program  Management  Approach  to  Services  Acquisitions 

The  lack  of  a  mature  program  management  infrastructure  and  a  life-cycle 
approach  to  services  acquisition  project  management  exposes  the  DoD  to  the  risk  of 
not  meeting  cost,  schedule,  and  performance  objectives  (Apte  &  Rendon,  2007). 

The  DoDIG  and  GAO  identified  critical  deficiencies  when  examining  the  DoD’s 
existing  management  structure  for  acquiring  services.  In  this  section,  we  review 
basic  program  management  concepts  and  the  implications  of  research  conducted  by 
Apte  and  Rendon  (2007),  which  examined  the  application  of  a  program  management 
approach  and  project  management  concepts  to  services  acquisition. 

1.  Overview  of  the  Program  Management  Approach  and  Project 
Management  Concepts 

Apte  and  Rendon  (2007)  use  the  term  program  management  to  describe  the 
approach  and  methodology  needed  for  the  management  of  complicated  projects.  A 
program  management  approach  includes  the  foundation  that  enables  the  attainment 
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of  cost,  schedule,  and  performance  objectives,  and  represents  the  coordinated 
centralized  management  of  multiple  projects  to  achieve  the  program’s  strategic 
objectives  and  benefits  (Apte  &  Rendon,  2007;  Project  Management  Institute  [PMI], 
2008).  A  systematic  program  management  approach  includes  the  following  basic 
project  management  concepts:  project  life  cycle,  integrated  processes,  project  team 
structure,  a  project  manager,  and  a  suitable  organizational  structure  (Apte  & 
Rendon,  2007).  These  project  management  concepts  are  examined  in  further  detail 
in  the  following  paragraphs. 

The  first  project  management  concept  we  examine  is  the  project  life  cycle. 
The  Project  Management  Institute  (PMI)  Project  Management  Body  of  Knowledge 
(PMBOK\  2008)  defines  project  life  cycle  as  a  collection  of  generally  sequential 
phases  whose  name  and  number  are  determined  by  the  control  needs  of  the 
organization(s)  involved  in  the  project.  By  dividing  the  project  into  phases, 
management  is  more  effectively  able  to  control  activities  within  each  phase  and  the 
overall  progress  of  the  project  (Apte  &  Rendon,  2007).  The  Defense  Acquisition 
Guidebook  (DAU,  2012)  breaks  the  project  life  cycle  into  the  following  phases: 
material  solution  analysis,  technology  development,  engineering  and  manufacturing 
development,  production  and  deployment,  and  operations  and  support. 

Integrated  processes  are  an  essential  element  of  a  systematic  program 
management  approach  and  are  vital  to  project  success.  The  PMI  PMBOK  (2008) 
identifies  five  project  management  process  groups  required  for  any  project.  These 
groups  are  initiating,  planning,  executing,  monitoring  and  controlling,  and  closing 
processes  (PMI,  2008).  Each  process  group  includes  functional  phases,  such  as 
cost  and  schedule  management,  that  are  part  of  a  respective  process  group  (PMI, 
2008).  A  structured  program  management  approach  integrates  these  processes  to 
ensure  coordination  and  unity  of  the  total  program  effort. 


The  project  team  structure,  with  cross-functional  expertise  across  various 
disciplines,  is  necessary  to  effectively  integrate  project  management  efforts  to 
achieve  the  project’s  objective.  In  collaboration  with  the  project  manager,  these 
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integrated  subject-matter  experts  must  coordinate  and  determine  which,  and  to  what 
degree,  respective  integrated  processes  are  appropriate  for  the  effort  (PMI,  2008). 

For  an  effective  program  management  approach,  a  project  manager  must  be 
designated  by  the  organization  to  synchronize  the  project  activities  of  the  various 
functional  team  members  towards  the  overall  project  objectives.  Complex  projects 
require  the  project  manager  to  oversee  activities  and  determine  applicable  resource 
applications  to  these  activities  (PMI,  2008). 

The  appropriate  organizational  structure  is  a  vital  element  of  a  disciplined 
program  management  approach.  Organizational  structures  influence  how  projects 
are  conducted  and  range  from  functional  to  project  specific,  with  various  matrix  types 
in  between  (PMI,  2008).  Selection  of  a  suitable  organizational  structure  that 
supports  the  integrated  processes,  project  teams,  and  project  manager  will 
substantially  contribute  to  project  success  (Apte  &  Rendon,  2007).  In  the  next 
section,  we  look  at  how  program  management  concepts  are  currently  applied  to 
weapons  systems  acquisitions. 

2.  Application  of  Program  Management  Concepts  to  Weapons 
Systems  Acquisitions 

Program  management  concepts  are  well  established  for  defense  weapons 
systems  acquisitions  and  are  essential  practices  for  complex  high-technology 
weapons  systems  projects  (Apte  &  Rendon,  2007).  The  defense  acquisition  life 
cycle  is  a  disciplined  management  process  that  takes  acquisition  programs  through 
a  series  of  phases,  milestones,  decision  points,  and  reports.  Control  gates  assist 
with  keeping  projects  within  the  three  major  constraints  of  cost,  schedule,  and 
performance  (Rendon  &  Snider,  2008).  DoD  Instruction  5000.2  (OUSD[AT&L],  2008) 
establishes  the  Defense  Acquisition  Management  System  as  the  project  life  cycle  for 
Major  Defense  Acquisition  Programs  and  was  updated  in  2008  to  specifically  include 
services.  (See  Figure  3  for  an  illustration  of  the  milestones  and  phases  of  the  DoD 
Acquisition  Management  Framework.)  The  program  manager  is  the  designated 
individual  responsible  for  program  objectives;  is  accountable  for  cost,  schedule,  and 
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performance;  and  reports  to  the  Milestone  Decision  Authority  (MDA;  OUSD[AT&L], 
2003).  Integrated  processes  and  integrated  product  teams  (IPTs),  which  are 
established  by  DoD  5000  regulations,  enable  program  managers  to  maintain 
continual  and  effective  communication  throughout  project  execution  (Apte  & 

Rendon,  2007).  The  DoD  relies  substantially  on  tailored  organizational  structures  to 
enhance  integration  of  project  processes  and  project  teams  for  weapons  systems 
acquisition  programs  (Apte  &  Rendon,  2007).  The  DoD  typically  uses  matrix  and 
projectized  (project  specific)  organizational  structures  for  weapons  systems 
acquisitions.  The  type  of  tailored  organizational  structure  used  for  a  specific  project 
depends  on  factors  such  as  the  number  of  functional  areas  involved  in  the  project, 
the  degree  of  integration  required  within  the  functional  areas  and  between  the 
organization  and  customer,  and  the  organization’s  experience  level  for  the  work  that 
the  project  requires  (Apte  &  Rendon,  2007).  Integrated  project  teams  are  cross¬ 
functional  teams  with  subject-matter  experts  from  multiple  functional  areas,  including 
engineering,  contracting,  financial  management,  logistics,  and  legal  (Apte  &  Rendon, 
2007).  These  teams  are  led  and  managed  by  the  designated  project  manager  who 
ensures  coordination  and  integration  to  accomplish  the  project’s  objective  (Apte  & 
Rendon,  2007).  In  the  next  section,  we  examine  how  these  proven  program 
management  concepts  for  weapons  systems  can  be  applied  to  services  acquisitions. 
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Figure  3.  The  Defense  Acquisition  Management  Framework 

(DAU,  2012) 
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3.  Application  of  Program  Management  Concepts  to  Services 
Acquisitions 

Apte  and  Rendon  (2007)  assessed  current  services  acquisition  practices  at 
various  activity  levels  of  the  Air  Force  and  examined  how  a  program  management 
approach  and  concepts  can  be  applied  to  successfully  manage  services  acquisition 
programs.  The  findings  showed  that  at  the  installation  level,  the  acquisition  of 
services  was  managed  ad  hoc  as  opposed  to  using  a  program  management 
approach  (Apte  &  Rendon,  2007).  The  Air  Force  used  some  program  management 
concepts  at  the  installation  level,  but  they  were  not  institutionalized  throughout  the 
organization  and  were  inconsistent  in  application  (Apte  &  Rendon,  2007).  Apte  and 
Rendon  (2007)  found  that  the  traditional  method  used  for  the  acquisition  of  services 
does  not  utilize  a  program  management  approach.  The  traditional  method  lacked 
the  disciplined  use  of  project  life  cycles,  integrated  processes,  designated  program 
managers,  integrated  cross-functional  teams,  and  an  appropriate  organizational 
structure. 

However,  Apte  and  Rendon  (2007)  observed  two  organizations  at  the  major- 
command  level  that  are  applying  innovative  program  management  approaches  in 
terms  of  organizational  structure,  project  life  cycle,  integrated  processes,  and  project 
teams  for  the  successful  acquisition  of  services.  The  Air  Education  and  Training 
Command  (AETC)  integrates  critical  processes  by  utilizing  a  disciplined  and 
structured  project  life  cycle  for  services  acquisitions  (Apte  &  Rendon,  2007).  Figure 
4  depicts  the  various  phases  within  the  AETC  services  project  life  cycle.  In  addition, 
the  AETC  utilizes  formal  project  teams  for  pre-  and  post-award  activities  and  an 
integrated  and  matrixed  organizational  structure  for  the  acquisition  of  support 
services  (Apte  &  Rendon,  2007).  Apte  and  Rendon  (2007)  found  the  Air  Combat 
Command  (ACC)  Acquisition  Management  and  Integration  Center  (AMIC)  to  be  a 
fully  integrated  organization  that  employs  the  critical  program  management  concepts 
discussed  previously  and  includes  all  essential  elements  of  an  acquisition  program 
office.  The  AMIC  provides  integrated  cradle-to-grave  services  acquisition  support, 
which  enables  a  resource-efficient  process-oriented  approach  for  acquisitions  (Apte 
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&  Rendon,  2007).  Both  the  AETC  and  the  AMIC  successfully  demonstrated  a 
disciplined  program  management  approach  for  the  acquisition  of  services  (Apte  & 
Rendon,  2007).  The  research  clearly  demonstrated  the  viability  and  applicability  of  a 
program  management  approach  to  services  acquisitions.  This  program 
management  approach  provides  a  process-oriented  framework  for  improved 
services  acquisition  outcomes. 

The  activities  and  steps  of  the  services  contracting  process  are  an  integral 
component  of  the  acquisition  life  cycle  and  are  discussed  in  the  next  section. 
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Figure  4.  Services  Project  Life  Cycle 

(Apte  &  Rendon,  2007) 

D.  Services  Contracting  Process 

Contract  management  is  an  integral  part  of  the  defense  acquisition 
management  system  and  is  defined  as  “the  art  and  science  of  managing  a 
contractual  agreement  throughout  the  contracting  process”  (Rendon  &  Snider, 

2008).  It  is  the  means  through  which  progress  is  made  in  the  different  phases  of  the 
acquisition  life  cycle.  The  acquisition  of  all  systems,  goods,  and  services  is  done 
through  contracting  and  is  guided  by  the  FAR.  The  contracting  process  involves  six 
primary  phases  conducted  by  both  a  buying  organization  (government)  and  a  selling 
organization  (contractor),  each  with  a  unique  contract  management  perspective  (see 
Figure  5  for  an  illustration).  The  buyer’s  perspective  involves  procurement  planning, 
solicitation  planning,  solicitation,  source  selection,  contract  administration,  and 
contract  closeout/termination  (Rendon  &  Snider,  2008).  The  seller’s  perspective 
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involves  similar  phases  that  align  with  the  buyer’s  phases.  They  are  pre-sales 
activities,  bid/no-bid  decision-making,  bid/proposal  preparation,  contract  negotiation 
and  formation,  contract  administration,  and  contract  closeout/termination  (Rendon  & 
Snider,  2008).  Since  this  research  is  about  the  problems  of  services  contracts  within 
the  U.S.  government,  we  discuss  only  the  buying  (government)  side  of  the 
contracting  process. 


Figure  5.  Procurement  Process 

(Rendon  &  Snider,  2008) 

Phase  1  for  the  buyer  is  the  procurement  planning  process  and  involves 
identifying  which  business  needs  can  be  met  by  reaching  outside  the  organization 
for  products  and  services  (Rendon  &  Snider,  2008).  The  government  decides  if 
procurement  is  really  necessary  and,  if  so,  what,  how  much,  and  when  to  procure 
(Rendon  &  Snider,  2008).  Part  of  the  planning  process  includes  determining  and 
defining  the  requirement  and  conducting  market  research  (Rendon  &  Snider,  2008). 
Budgets  and  cost  estimates  are  developed  by  both  the  buyer  and  seller,  and 
preliminary  work  documents  that  delineate  what  services  will  be  performed  are 
written  (Rendon  &  Snider,  2008). 

Phase  2  is  solicitation  planning.  The  process  involves  determining  which 
procurement  method  the  government  will  use  to  procure  the  goods  or  services,  such 


NPS 


prMVTW^TIA  PER 

MI1I1IJ1  - - / 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  17  - 


as  through  negotiated  proposals  or  sealed  bids.  The  type  of  contract  (cost  versus 
fixed  price)  and  the  contract/award  strategy  (lowest  price  versus  price  trade-off)  are 
determined,  criteria  are  developed  for  evaluating  proposals,  and  the  solicitation 
documentation  is  drawn  up  with  contract  terms  and  conditions  (Rendon  &  Snider, 
2008). 


Phase  3  consists  of  the  solicitation  process.  Solicitation  involves  receiving 
bids  and  proposals  from  potential  sellers  (contractors).  In  order  to  receive  bids  and 
increase  competition,  the  government  must  advertise  that  there  is  a  procurement 
opportunity  (Rendon  &  Snider,  2008).  To  do  this,  the  contracting  officer  transmits  a 
notice  though  the  Government  Point  of  Entry  (Rendon  &  Snider,  2008).  After  the 
government  issues  the  solicitation,  contractors  prepare  offers  and  attempt  to 
persuade  the  government  to  accept  their  bid  (Rendon  &  Snider,  2008). 

Phase  4  is  the  source  selection  and  involves  receiving  all  the  bids  or 
proposals  from  the  sellers.  The  proposals  are  then  evaluated  against  the  criteria 
previously  set  in  the  solicitation  planning  phase  (Rendon  &  Snider,  2008).  Before 
the  contract  can  be  awarded,  contract  negotiations  must  occur  between  the 
government  and  the  contractor  to  ensure  that  there  is  full  agreement  on  cost, 
schedule,  terms  and  conditions,  and  contractor  performance  (Rendon  &  Snider, 
2008). 


Phase  5,  contract  administration,  is  the  most  critical  phase  in  the  contracting 
process.  Both  the  government  and  the  contractor  conduct  contract  administration, 
each  ensuring  that  performance  from  both  sides  meets  the  contractual  obligations. 
Both  parties  attend  a  pre-performance  conference  to  ensure  that  everyone 
understands  the  contract  requirements  and  to  discuss  protocols  for  performance 
management,  communication,  and  contract  change  management  (Garrett  & 

Rendon,  2005).  Any  changes  or  modifications  to  the  contract  are  executed,  by 
authorized  individuals  only,  through  a  formal  process  (Rendon  &  Snider,  2008).  The 
government  closely  monitors  the  contractor  for  quality,  cost  control,  and 
performance.  Best  practices  in  contract  administration  include,  but  are  not  limited  to, 
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establishing  a  system  to  verify  contractual  conformance,  assigning  responsibility  to 
check  actual  performance  against  requirements,  ensuring  that  someone  takes 
appropriate  corrective  action  to  significant  variances,  and  maintaining  all  contract 
documentation  (Garrett,  2011). 

Phase  6  is  contract  closeout  or  termination.  A  government  contract  can  end 
when  the  contractor  successfully  completes  the  job,  when  the  government  deems 
that  it  is  convenient  to  terminate  the  contract,  or  when  the  contractor  is  in  default 
(Rendon  &  Snider,  2008).  Regardless  of  why  the  contract  is  closed,  all  final 
administrative  and  legal  matters  must  be  completed  (e.g.,  price  adjustments  and 
final  payments  made,  claims  settled,  final  products  or  services  accepted,  and  past- 
performance  reports  documented;  Rendon  &  Snider,  2008). 

Recent  studies  on  the  contract  management  process  in  the  DoD  have 
indicated  that,  on  the  average,  the  process  capability  for  the  procurement  planning, 
solicitation  planning,  solicitation,  and  source  selection  phases  are  at  the  Structured 
level  (Level  3),  meaning  that  these  processes  are  not  fully  integrated  throughout  the 
other  functional  areas  of  the  agency,  even  though  they  are  institutionalized.  They 
are  also  not  measured  or  continuously  improved.  Additionally,  on  the  average,  the 
process  maturity  level  for  the  contract  administration  and  contract  closeout  phases 
are  at  the  Basic  level  (Level  2),  meaning  that  these  processes  are  not  sufficiently 
institutionalized  within  the  contracting  department,  even  though  they  are  somewhat 
established  with  some  documentation,  nor  are  these  processes  fully  integrated 
throughout  the  other  functional  areas  of  the  agency  (Rendon,  2009,  201 0,  201 1 ). 

Each  of  the  phases  in  the  contracting  process  has  activities  performed  by 
different  individuals,  each  of  whom  has  a  stake  in  making  sure  that  a 
project/program  is  performed  in  accordance  with  their  organization’s  regulations. 
These  stakeholders  can  affect,  or  can  be  affected  by,  the  accomplishment  of  their 
organization’s  objectives  (Freeman,  1984).  Stakeholder  Theory  describes  this 
concept  and  is  discussed  next. 
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E.  Stakeholder  Theory 


In  the  private  sector,  Stakeholder  Theory  is  described  as  a  corporation  that 
identifies  the  people  who  have  a  stake  or  interest  in  that  corporation  and  then  acts 
accordingly  to  further  the  interests  of  those  stakeholders  (Cleland,  1 986).  The  main 
assumption  of  this  theory  is  that  the  purpose  of  the  corporation  is  to  generate  and 
dispense  some  form  of  wealth  to  various  stakeholders,  and  that  in  order  to  achieve 
that  purpose,  all  of  the  stakeholders  cooperate  (Freeman,  1984).  The  word 
stakeholder  originated  in  the  1960s  at  the  Stanford  Research  Institute  (SRI)  with 
respect  to  the  view  that  corporate  management  should  only  be  concerned  with  the 
corporation’s  stakeholders  (Parmar,  Freeman,  Harrison,  Wicks,  Purnell,  &  De  Colle, 
2010).  Figure  6  illustrates  Stakeholder  Theory.  This  theory,  applied  to  government 
contracting,  includes  all  stakeholders  concerned  with  a  certain  project.  The  project 
is  the  corporation,  and  the  contractor  and  the  various  government  roles  on  the 
acquisition  team  are  the  stakeholders.  That  is  a  very  broad  view  because  both  the 
contractor  and  the  government  have  numerous  individual  stakeholders.  On  the 
government  side,  there  is  the  contracting  officer  (CO),  the  program  manager  (PM), 
the  contracting  officer  representative  (COR),  the  financial  managers,  and  the 
customers  who  use  the  final  product.  On  the  contractor  side,  there  is  the  contracts 
manager,  the  contract  specialist,  the  program  manager,  all  subcontractors 
conducting  business  with  that  company,  all  suppliers  who  provide  goods  or  services 
to  that  company,  and  various  executives  who  have  a  vested  interest  in  the  success 
of  the  company.  This  is  a  serious  challenge  for  most  managers  because  the 
stakeholder  management  approach  views  success  as  assessing  the  impact  of  every 
decision  made  on  all  stakeholders  through  the  services  acquisition  process  (Cleland, 
1986). 
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Figure  6.  A  Schematic  Model  Illustrating  the  Proposed  Stakeholder 

Theory  of  the  Organization 

(Parmar  et  al. ,  2010) 

As  stated  earlier,  the  CO,  PM,  COR,  financial  manager,  and  customer  are  all 
government  stakeholders  on  various  projects  that  fall  under  their  sphere  of  influence. 
As  a  result  of  their  differing  roles  and  responsibilities,  their  definitions  of  success 
cover  a  wide  range  and  are  often  in  conflict  with  each  other  in  terms  of  importance  of 
the  factors  measured.  These  stakeholders  are  described  in  detail  in  the  next  section. 


F.  Stakeholders  in  DoD  Services  Contracts 

1.  Contracting  Officer 

The  government  obtains  defense  systems,  equipment,  and  services  from 
private-sector  sources.  Since  a  contract  is  the  primary  means  of  acquisition, 
contract  management  is  a  set  of  important  skills  and  knowledge  that  is  of  great  value 
in  defense  acquisition  projects  (Rendon  &  Snider,  2008).  The  FAR  (2012)  defines 
the  CO  as  an  agent  of  the  government  with  the  authority  to  enter  into,  administer, 
and/or  terminate  contracts  and  make  related  determinations  and  findings.  Not  only  is 
the  CO  responsible  for  performing  all  contract  functions,  but  the  CO  must  also  be 
able  to  aid  in  requirements  development  (DoD,  2011).  The  CO  is  the  principal 
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business  advisor  to  the  acquisition  team,  and  the  role’s  specific  responsibilities 
include  forming  a  business  strategy,  participating  in  source  selection,  and  possibly 
administering  the  resultant  contract  (DoD,  2011).  He  or  she  is  the  individual 
responsible  for  making  sure  that  contracts  are  planned,  executed,  and  closed  out  in 
accordance  with  agency  regulations  and  statutory  requirements  (Rendon  &  Snider, 
2008).  COs  must  provide  support  to  help  achieve  the  contract  objectives  of  cost, 
schedule,  and  performance.  COs  are  the  only  agents  with  the  ability  to  bind  the 
government,  and  any  individual  who  attempts  to  do  so  without  delegated  authority 
will  make  an  unauthorized  commitment  (Rendon  &  Snider,  2008).  A  detailed  listing 
of  contracting  officer  responsibilities  is  presented  in  the  FAR  (2012). 

2.  Program/Project  Manager 

The  PM,  also  called  a  program  director  or  a  project  manager,  is  the 
designated  person  responsible  for  accomplishing  program  objectives  and  ensuring 
that  the  desired  results  are  achieved  (Brown,  2010).  The  PM  has  a  very  involved 
and  important  role  in  acquisition  and  procurement  because  he  or  she  is  held 
accountable  for  overall  cost,  schedule,  and  performance.  The  PM  identifies,  plans, 
and  controls  various  aspects  of  the  project/program.  These  areas  include,  but  are 
not  limited  to,  delivery  requirements,  scheduling,  conducting  market  research,  and, 
normally,  participating  in  source  selection  (DoD,  2011).  The  PM  should  be  the  one 
most  familiar  with  the  program  requirements  (DoD,  2011). 

The  PM  must  also  have  aptitude  in  contracting,  financial  management,  and 
cost  estimating.  And,  more  important,  his  or  her  management  and  leadership  skills 
come  into  play  when  dealing  with  management  challenges  in  day-to-day  program 
execution  (Wood,  2010).  The  PM’s  role  is  not  an  easy  one,  and  the  GAO  has 
historically  identified  his  or  her  lack  of  program  management  expertise  as  the 
primary  cause  of  cost  and  schedule  overruns  in  major  acquisitions  (GAO,  2005). 
Major  weapons  systems  typically  establish  a  program  office;  however,  this  is  not 
always  the  case  with  services  acquisitions  (GAO,  2011a).  When  there  is  no  PM 
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designated  on  a  services  contract,  the  contracting  organization  will  have  to  work 
directly  with  the  requiring  organization  (GAO,  2011a). 

3.  Contracting  Officer  Representative 

Complex  contracts  for  services  require  the  addition  of  a  technical  expert,  one 
who  is  intimately  involved  in  the  contract  and  has  expert  technical  knowledge  of  the 
system  or  service  being  procured — this  person  is  the  COR.  The  COR  is  the  onsite 
technical  specialist  who  assesses  performance  against  standards  and  then  records 
and  reports  this  information  to  the  CO  (GAO,  201  la).  Only  a  U.S.  government 
employee  (civilian  or  military)  or  a  North  Atlantic  Treaty  Organization  (NATO)  partner 
can  fill  the  role  of  a  COR;  the  CO  should  never  fill  this  role  (FAR,  2012).  The  COR  is 
formally  appointed  in  writing  by  the  contracting  officer  and  must  have  specific 
qualifications  and  experience  appropriate  for  the  responsibility  delegated  to  them 
(FAR,  2012).  The  COR  should  be  deeply  involved  in  the  entire  acquisition  and 
procurement  process  and  will  assist  the  contracting  officer  in  developing  the  quality 
assurance  plan,  the  technical  requirements  in  the  contract,  and  other  pre-award 
activities.  The  COR  does  not  have  the  same  authority  as  the  contracting  officer,  so 
he  or  she  cannot  make  any  commitments  or  changes  that  affect  the  terms  of  the 
contract  (FAR,  2012).  The  COR  assists  the  contracting  officer  and,  along  with  the 
PM,  becomes  the  focal  point  of  the  contract  by  monitoring  all  of  its  day-to-day 
aspects,  as  well  as  inspecting  and  accepting  services  (DAU,  2012).  A 
comprehensive  list  of  the  specific  responsibilities  of  a  COR  can  be  found  in  the  FAR 
(2012). 

4.  Finance  Manager 

The  finance  manager  is  another  critical  individual  in  contract  management 
and  is  well  versed  in  the  financial  management  regulation  (FMR).  His  or  her  role  is  to 
serve  as  the  fiscal  and  budgetary  advisor  to  the  acquisition  team  (DoD,  2011).  The 
government  finance  manager  is  responsible  for  ensuring  compliance  with  the 
statutory  requirements  of  fiscal  law  (e.g.,  that  proper  authorization  is  granted  for 
expending  funds  [purpose],  the  contract  obligations  occur  during  the  time  limits 

ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  23  - 

NAVAL  POSTGRADUATE  SCHOOL 


prescribed  by  appropriation  [time],  and  adequate  funding  is  available  [amount]; 
Rendon  &  Snider,  2008). 

5.  Customer  or  End  User 

The  customer’s  role  in  the  acquisition  process  is  to  have  a  detailed 
knowledge  of  the  requirement  so  that  it  can  be  clearly  conveyed  to  contractors  (DoD, 
2011).  The  customer  helps  to  determine  whether  trade-offs  are  available  for  a 
requirement  and  what  these  trade-offs  are  (DoD,  201 1 ).  The  customer  plays  a  vital 
role  in  the  acquisition  process  because,  ultimately,  he  or  she  is  a  member  of  the 
team  that  drives  how  well  the  requirements  document  reflects  their  needs. 

The  roles  and  responsibilities  of  each  of  the  stakeholders  differ,  and  as  a 
result,  their  definitions  of  what  constitutes  a  successful  contract  cover  a  wide  range. 
Often  these  definitions  are  in  conflict  with  each  other  in  terms  of  the  importance  of 
the  factors  measured.  Identifying  how  each  of  the  stakeholders  determines  success 
is  vital  to  contract  administration.  In  the  next  section,  we  discuss  a  study  that 
identified  seven  factors  that  contribute  to  contract  management  success. 

G.  Success  Factors 

As  described  earlier  in  this  chapter,  there  are  six  phases  in  the  contract 
management  process.  Of  these  six  phases,  only  one  is  dedicated  to  contract 
administration.  GAO  audits  have  consistently  stated  that  contract  administration 
needs  to  be  improved.  Successful  contract  administration  is  dependent  on  effective 
contract  management.  The  question  then  becomes,  what  determines  the  success  of 
a  contract?  Rendon  (2012)  conducted  a  study  asking  this  question  specifically  of 
DoD  contracting.  In  the  study,  Rendon  surveyed  eight  defense  agencies  and  two 
defense  contractors  over  the  course  of  four  years.  Since  the  survey  was  designed 
for  defense  procurement,  it  was  administered  only  to  warranted  contracting  officers 
and  individuals  fully  qualified  in  government  contracting.  The  results  of  the  survey 
identified  seven  critical  success  factors  (CSF)  in  defense  procurement:  Workforce, 
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Relationships,  Processes,  Resources,  Leadership,  Policy,  and  Requirements 
(Rendon,  2012). 

Rendon  analyzed  and  summarized  over  2,000  responses  to  the  survey  into 
the  seven  CSF  categories.  The  Workforce  factor  in  the  study  related  to  using  proper 
staffing,  hiring,  and  recruitment  processes;  having  the  right  number  of  personnel; 
and  having  experienced,  trained,  and  competent  people.  Relationship  responses 
involved  communication,  cooperation,  and  coordination  at  all  levels  within  and 
between  agencies.  The  Processes  category  involved  having  a  consistent,  efficient, 
standardized,  enforced,  streamlined,  and  documented  contracting  process. 

Resource  responses  included  the  need  for  contract  tracking  tools,  automated 
contract  writing  systems,  technical  support,  and  adequate  travel  funds.  Leadership 
responses  related  to  the  need  for  clear  lines  of  authority,  strong  management 
support,  and  an  empowered  leadership.  The  Policies  category  included  the  need  for 
clear  and  concise  guidance  and  regulations.  The  final  CSF,  Requirements,  related 
to  timely  procurement  request  packages,  well-written  statements  of  work,  proper 
technical  reviews,  and  adequate  procurement  funding  (Rendon,  2012). 

A  comparison  of  the  DoD  and  industry/contractor  responses  shows  some 
interesting  differences.  The  DoD  considers  Workforce-related  elements  to  be  the 
most  important  success  factor  and  Requirements-related  elements  to  be  the  least 
important.  Industry  responses  showed  that  Processes  were  the  most  important 
factor  and  Policies  were  the  least  important.  The  overall  results  of  the  study  provide 
some  thought-provoking  insights  into  the  differences  between  organizations  and  the 
disconnection  that  can  occur  when  measuring  and  defining  the  success  of  a 
contract.  Agency  Theory,  described  in  the  next  paragraph,  explains  how  and  why 
this  disconnection  can  occur. 

H.  Agency  Theory 

Agency  Theory  is  aimed  at  the  relationship  that  arises  when  one  party  (the 
principal)  engages  another  party  (the  agent)  to  perform  a  specific  effort  focusing  on 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  25  - 

NAVAL  POSTGRADUATE  SCHOOL 


a  certain  outcome  (Eisenhardt,  1989).  The  theory  mainly  discusses  these 
relationships  and  describes  how  the  principal  and  the  agent  are  engaged  in 
cooperative  behavior,  but  have  differing  goals  and  attitudes  (Eisenhardt,  1989). 
Agency  Theory  states  that  the  principal  can  limit  this  difference  by  creating  suitable 
incentives  for  the  agent  or  creating  sufficient  means  to  monitor  the  agent  (Hill  & 
Jones,  1992).  When  applying  this  theory  to  services  contracts,  the  government  is 
the  principal  and  the  contractor  is  the  agent.  The  government  employs  the 
contractor  to  perform  specific  tasks  that  are  defined  in  great  detail  within  the 
contract.  As  described  in  the  previous  section,  the  government  and  contractor  place 
differing  importance  on  each  of  the  success  factors — and  this  is  one  of  the 
cornerstone  assumptions  of  Agency  Theory  (Hill  &  Jones,  1992).  The  government 
and  the  contractor  clearly  have  different  goals  in  mind  when  executing  a  services 
contract.  The  government’s  goal  is  to  have  the  services  performed  to  a  certain 
standard  in  the  most  effective  way  for  a  fair  and  reasonable  price  (FAR,  2012).  The 
contractor  wishes  to  stay  in  business  so  its  ultimate  goal  is  to  generate  profit.  The 
government  incentivizes  the  contractor  by  providing  a  fee — and  in  some  cases,  a 
price  premium — in  order  to  facilitate  desired  behaviors. 

Agency  Theory  describes  the  complex  government-contractor  relationship 
and  how  a  principal  and  an  agent  can  be  involved  in  a  cooperative  effort  but  have 
differing  goals  and  attitudes  (Eisenhardt,  1989).  In  the  next  section,  we  summarize 
this  literature  review  and  build  a  foundation  for  Chapter  III. 

I.  Summary 

This  chapter  introduced  past  and  current  literature  on  the  deficiencies  in 
services  contracting,  the  need  for  a  program  management  approach  to  the  services 
contracting  process,  the  current  services  contracting  process,  Stakeholder  Theory 
and  the  various  stakeholders  on  the  acquisition  team,  critical  success  factors  in 
government  contract  management,  and  the  application  of  Agency  Theory  to  the 
services  contracting  process. 
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The  next  chapter  describes  our  research  methodology,  including  the  data 
collection  process,  a  description  of  the  participant  commands,  a  list  of  the 
demographic  questions  asked,  a  brief  description  of  the  survey  questions  related  to 
the  core  research,  and  the  type  of  analysis  we  conducted. 
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Research  Methodology 


A.  Introduction 

In  this  chapter,  we  explain  how  we  collected  and  analyzed  our  data  in  order  to 
meet  our  objectives  and  answer  the  research  questions  discussed  in  Chapter  I. 
Specifically,  we  discuss  the  creation  of  the  survey  and  our  analytical  process.  We 
also  include  a  description  of  the  quantitative  methods  we  used  to  analyze  the  data 
collected  from  the  Navy  contracting  commands.  The  objective  of  the  research  was 
to  build  upon  the  results  found  in  a  prior  study,  so  that  we  could  conduct  a  more  in- 
depth  research  analysis  that  further  explains  the  factors  that  influence  the  success  of 
services  contracts.  We  analyze  the  collected  data  quantitatively  in  order  to  draw 
conclusions  about  the  definition  and  measurement  of  the  success  of  services 
contracts,  and  compare  the  similarities  and  differences  among  the  stakeholders 
surveyed. 

B.  Overview 

Our  research  methodology  included  development  of  a  survey  instrument  to 
collect  empirical  data  for  answering  our  research  questions.  The  survey  was 
deployed  to  the  various  stakeholders  at  the  participating  commands.  We  then 
analyzed  the  data  using  descriptive  statistics  to  provide  recommendations  and 
conclusions. 

C.  Survey  Development 

Under  the  guidance  of  Professors  Apte  and  Rendon,  we  developed  a  web- 
based  survey  using  the  survey  software  Survey  Monkey.  We  first  tested  the  survey 
on  Naval  Postgraduate  School  (NPS)  contracting  students  for  validity  and 
cohesiveness,  and  used  the  survey  to  collect  empirical  data  regarding  the  definition 
and  measurement  of  services  contracts.  The  focus  of  the  survey  was  on  answering 
the  following  core  research  questions: 
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How  do  different  stakeholders  define  successful  services  contracts 
within  the  Navy? 


■  How  do  different  stakeholders  measure  services  contracts  within  the 
Navy? 

We  took  the  qualitative  results  from  previous  research  conducted  by  Miller, 
Newton,  and  D’Amato  (2012)  and  then  identified  four  metrics — process,  cost, 
schedule,  performance — about  which  we  could  ask  further  detailed  questions. 


The  survey  questions  consisted  of  seven  demographic  questions  and  12 
research  questions  that  addressed  our  two  core  research  questions: 


1 .  What  is  your  branch  of  Service  or  Service  affiliation? 

2.  What  organization  are  you  affiliated  with? 

3.  What  is  your  current,  primary,  functional  role? 

4.  What  is  your  DAWIA  level  certification? 

5.  How  many  years  of  acquisition  experience  do  you  have? 

6.  What  type  of  services  do  you  predominantly  procure? 

7.  What  broad  category  do  the  majority  of  your  contracts  fall  into? 

8.  How  do  you  define  a  successful  service  contract? 

9.  How  do  you  measure  the  success  of  a  service  contract? 

The  purpose  of  the  demographic  questions  was  so  that  we  could  differentiate 
our  results  and  compare  and  contrast  to  determine  trends  across  different  areas, 
such  as  functional  role,  DAWIA  level,  and  type  of  service  provided. 


The  purpose  of  the  core  research  questions  was  to  establish  the  importance 
of  different  factors  when  defining  and  measuring  the  success  of  services  contracts. 
We  asked  several  questions  related  to  the  contracting  process,  as  well  as  questions 
concerning  different  outcomes  such  as  cost,  schedule,  and  performance.  Our 
process  questions  involved,  but  were  not  limited  to,  the  level  of  administrative  load, 
occurrence  of  protests,  and  levels  of  communication  between  all  stakeholders.  The 
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survey  questions  associated  with  cost  dealt  with  overruns,  fair  and  reasonable 
pricing  and  profit,  as  well  as  cost  control.  The  schedule  questions  were  related  to 
meeting  major  milestones  and  a  timely  completion  of  the  contract.  Performance 
questions  were  connected  to  customer  satisfaction,  adherence  to  the  statement  of 
work  (SOW),  and  reliance  on  COR  reports.  We  also  used  these  factors  to 
differentiate  responses  in  order  to  determine  trends. 

The  survey  provides  specific  questions  related  to  how  commands  define  the 
success  of  a  services  contract.  The  first  two  questions  ask  participants  to  rank 
various  definitions  relating  to  the  four  metrics  in  order  of  most  important  (1)  to  least 
important  (5).  The  next  three  main  questions  ask  participants  to  rate  definition 
statements  relating  to  process,  schedule,  cost,  and  performance.  These  questions 
use  a  Likert  scale  asking  level  of  agreement,  importance,  and  amount  of  time 
devoted  by  the  participants.  The  Likert  scale  had  a  range  of  (1 )  to  (5),  with  (1 ) 
representing  a  negative  response  and  (5)  representing  a  positive  response. 

The  survey  also  asks  specific  questions  related  to  how  commands  measure 
the  success  of  a  services  contract.  The  first  two  questions  ask  participants  to  rank 
various  measurements  relating  to  the  four  metrics  in  order  of  most  important  (1)  to 
least  important  (5).  The  last  question  in  the  section  asks  participants  to  rate  on  a 
Likert  scale  how  often  the  organization  conducts  certain  actions  that  pertain  to  the 
measurement  of  success  concerning  process,  schedule,  cost,  and  performance. 

Finally,  the  survey  includes  a  final  question  soliciting  any  general  comments 
that  the  participants  may  wish  to  share  regarding  the  topic  of  defining  and  measuring 
successful  services  contracts.  Figure  7  contains  a  diagram  of  the  survey  questions. 
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Figure  7.  Diagram  of  Survey  Questions 
D.  Survey  Deployment 

We  conducted  a  survey  with  the  major  stakeholders  in  the  services 
acquisition  process.  This  included  the  PMs,  COs,  CORs,  contractors,  and  end  users 
associated  with  the  following  contracting  commands:  the  Fleet  Logistics  Center 
(FLC)  Philadelphia,  FLC  Jacksonville,  FLC  Norfolk,  FLC  Puget  Sound,  FLC  San 
Diego,  Naval  Sea  Systems  Command  (NAVSEA),  Military  Sealift  Command  (MSC), 
and  Space  and  Naval  Warfare  Systems  Command  (SPAWAR). 

1.  Participating  Commands 

NAVSEA’s  mission  is  to  engineer,  build,  buy,  and  maintain  ships,  submarines, 
and  their  combat  systems  that  meet  the  United  States  Naval  Fleet’s  current  and 
future  operational  requirements.  As  the  largest  of  the  Navy’s  five  system 
commands,  the  NAVSEA  has  a  $30  billion  fiscal  year  budget,  accounting  for  25%  of 
the  Navy’s  entire  budget.  To  accomplish  their  mission,  the  NAVSEA  manages 
billions  of  dollars  in  annual  foreign  military  sales  and  150  acquisition  programs.  It 
plays  a  critical  role  in  the  Navy  Enterprise  and  strives  to  be  an  efficient  provider  of 
defense  resources  for  the  U.S.  (NAVSEA,  n.d.). 
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SPAWAR  is  one  of  three  major  Department  of  Navy  acquisition  commands. 

Its  mission  is  to  design,  develop,  and  deliver  advanced  communications  and 
information  dominance  systems  to  the  fleet.  It  supports  the  full  life  cycle  of  product 
and  services  delivery,  which  includes  research,  engineering,  acquisition  and 
deployment,  and  operations  and  logistics  support  services.  The  SPAWAR’s  products 
and  services  “transform  ships,  aircraft  and  vehicles  from  individual  platforms  into 
integrated  battle  forces,  enhancing  information  dominance  and  awareness  among 
Navy,  Marine,  joint  forces,  federal  agencies  and  international  allies”  (SPAWAR, 
n.d.).  In  order  to  accomplish  their  mission,  SPAWAR  partners  and  contracts  with 
industry,  including  small  businesses,  to  get  the  best  value  for  information 
technology.  In  2010,  they  obligated  $1 .21  billion  to  small  businesses  and  $4.84 
billion  to  large  businesses  (Esaias,  2011). 

The  MSC  is  composed  of  over  100  non-combatant,  civilian-crewed  ships.  Its 
mission  is  to  support  our  nation  by  replenishing  U.S.  naval  ships,  moving  military 
cargo  and  supplies,  strategically  prepositioning  combat  cargo  at  sea  around  the 
world,  and  conducting  specialized  missions.  The  MSC  is  organized  around  five 
mission  areas:  combat  logistics  force,  special  mission,  propositioning,  services 
support,  and  sealift.  These  worldwide  operations  are  financed  through  the  Navy 
Working  Capital  Fund  and  the  Transportation  Working  Capital  Fund.  Their  budget  of 
about  $3  billion  is  reimbursed  by  direct  appropriations  or  by  funds  transfers  by  MSC 
customers  (www.msc.navy.mil). 

The  FLCs  Philadelphia,  Jacksonville,  Norfolk,  Puget  Sound,  and  San  Diego 
are  just  some  of  the  12  total  subordinate  commands  of  the  Naval  Supply  Systems 
Command  (NAVSUP).  NAVSUP  as  a  whole  provides  25  distinct  products  and 
services,  ranging  from  supply  chain  management,  warehousing,  and  foreign  military 
sales,  to  postal  services  and  quality  of  life  programs  (NAVSUP,  n.d.).  Each 
subordinate  command  is  responsible  for  providing  logistics,  business  and  support 
services,  and  products  to  United  States  naval  activities  and  other  joint,  civilian,  and 
allied  forces  within  their  area  of  responsibility.  They  “deliver  combat  capability 
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through  logistics  by  teaming  with  regional  partners  and  customers  to  provide  supply 
chain  management,  procurement,  contracting  and  transportation  services,  technical 
and  customer  support,  defense  fuel  products  and  worldwide  movement  of  personal 
property”  (NAVSUP,  n.d.).  They  are  given  contracting  authority  by  NAVSUP, 
contracting  for  over  $4  billion  annually  in  supplies,  services,  and  equipment,  and 
make  about  120,000  individual  purchases  (GlobalSecurity.org,  n.d.). 

E.  Analytical  Process 

We  chose  ranking  and  Likert  scale  questions  to  gain  quantitative  data  for  our 
research  and  to  narrow  down  responses  from  various  stakeholders.  We  reported 
and  analyzed  the  mean  of  Likert  scale  responses  and  all  responses  that  received  a 
rank  of  (1 )  or  (2).  We  examined  the  data  to  determine  trends  and  identify 
correlations.  Initially,  we  took  an  overarching  view  of  the  responses,  including  all 
stakeholders  across  all  demographics.  We  conducted  further  analysis  by  breaking 
down  the  demographics  into  major  stakeholders,  DAWIA  level  certification,  and  type 
of  service  procured.  In  Chapter  IV,  we  present  this  quantitative  data  in  graphical 
format. 

F.  Summary 

In  Chapter  III,  we  identified  the  Navy  contracting  organizations  we  surveyed, 
how  we  created  the  survey,  and  how  we  collected  and  analyzed  data.  In  Chapter  IV, 
we  present  the  results  of  the  data  and  conduct  an  analysis  of  the  findings,  and  in 
Chapter  V  we  draw  conclusions  and  make  recommendations. 
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IV. 


Survey  Instrument,  Results,  and  Analysis 


A.  Introduction 

In  this  chapter,  we  examine  the  survey  responses.  The  objective  of  this 
research  was  to  examine  different  stakeholders’  definitions  and  measurements  of 
successful  services  contracts.  We  designed  a  survey  of  19  questions  and  presented 
them  to  the  major  stakeholders  in  the  services  acquisition  process.  The  survey  was 
distributed  to  the  following  activities:  NAVSEA,  SPAWAR,  MSC,  FLC  Philadelphia, 
FLC  Jacksonville,  FLC  Norfolk,  FLC  Puget  Sound,  and  FLC  San  Diego.  Our 
response  rate  is  shown  in  Table  1 .  We  received  such  a  small  number  of  responses 
from  requirements  managers,  financial  managers,  contractors,  and  customers  that 
their  responses  were  not  incorporated  in  this  report  for  analysis  purposes.  These 
respondents  are  combined  in  the  “other”  category  in  Table  1 . 


Table  1.  Survey  Response  Rate 


STAKEHOLDER 

#  SURVEYS 
DEPLOYED 

#  SURVEYS 
ANSWERED 

RESPONSE 

RATE 

PROGRAM  MANAGER/PROJECT  OFFICER 

94 

15 

16% 

CONTRACTING  OFFICER 

REPRESENTATIVE 

104 

27 

26% 

CONTRACTING  OFFICER/  CONTRACT 
SPECIALIST 

280 

126 

45% 

AGGREGATE  (PM,  COR,  PCO) 

478 

168 

35% 

OTHER 

365 

10 

2.7% 

TOTAL 

843 

178 

21% 

When  we  examine  the  ranking  questions  in  this  section,  the  term  “most 
important”  refers  to  the  number  of  factors  that  received  the  ranking  of  (1 )  or  (2).  We 
felt  that  this  was  the  most  accurate  way  to  capture  the  nature  of  our  participants’ 
responses.  For  example,  a  COR  may  feel  that  performance  factors  should  be 
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ranked  (1 )  every  time,  but  they  feel  that  process  factors  are  also  very  important  so 
they  may  have  responded  with  a  rank  of  (2).  We  believe  that  this  is  the  most 
effective  way  to  report  and  analyze  the  results  when  ranking  is  concerned. 

B.  Aggregate  Findings 

1.  Defining  the  Success  of  a  Service  Contract 

We  first  took  an  overarching  view  of  our  survey  findings.  We  did  not 
differentiate  between  functional  roles,  DAWIA  levels  of  certification,  contract  type,  or 
organization.  However,  we  did  separate  our  findings  under  the  broad  categories  of 
process  and  outcome.  Outcome  results  included  the  questions  associated  with  cost, 
schedule,  and  performance.  Our  survey  resulted  in  168  respondents,  spanning 
major  stakeholders.  The  Likert  scale  responses  were  assigned  a  value  of  (1 )  through 
(5),  with  the  higher  value  representing  a  favorable  response.  We  examined  the 
means  of  each  set  of  Likert  scale-type  questions  and  found  that,  when  defining  the 
success  of  a  services  contract,  outcomes  are  considered  slightly  more  important 
than  processes.  The  overall  mean  with  relation  to  outcomes  was  4.08,  while 
process  responses  produced  a  mean  of  3.97.  Our  findings  are  displayed  graphically 
in  Figure  8. 

We  then  separated  our  findings  further  within  the  broad  category  of  outcomes 
to  the  narrower  categories  of  cost,  schedule,  and  performance.  Performance-related 
questions  resulted  in  the  highest  mean,  4.29,  while  cost-related  questions  produced 
a  mean  of  4.03,  and  schedule-related  questions  produced  a  mean  of  3.93. 

One  hundred  and  sixty-eight  respondents  were  asked  to  rank  different  factors 
related  to  defining  the  success  of  a  service  contract.  These  questions  also  dealt 
with  different  aspects  of  processes  and  outcomes.  Of  the  172  respondents,  40%  felt 
that  process-related  factors  were  the  most  important.  Sixty  percent  felt  that 
outcome-related  factors  were  the  most  important.  The  distribution  of  highest  ranked 
responses  is  displayed  in  Figure  9. 
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Breaking  down  the  outcome-related  factors  further,  15%  of  respondents  felt 
that  cost-related  factors  were  the  most  important,  19%  felt  that  schedule-related 
factors  were  most  important,  and  26%  felt  that  performance-related  factors  were 
most  important. 
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Figure  8.  Means  of  Aggregate  Stakeholder  Definitions  of  Success 
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Figure  9.  Aggregate  Stakeholder  Ranking  of  Definitions  of  Success 
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2.  Measuring  the  Success  of  a  Service  Contract 

Our  survey  also  had  participants  rate  on  the  Likert  scale  the  various  degrees 
of  importance,  and  the  extent  to  which  they  agreed  or  disagreed  with,  various  factors 
related  to  how  they  measure  the  success  of  a  service  contract.  Again,  these  factors 
related  to  either  processes  or  outcomes.  The  overall  Likert  scale  mean  for  all 
demographics  with  relation  to  processes  was  2.48,  and  outcomes  displayed  an 
overall  mean  of  3.7 1 .  Clearly,  outcomes  are  more  important  to  our  participants  as  a 
whole.  Our  findings  are  displayed  graphically  in  Figure  10. 

If  we  look  at  the  distinct  factors  within  the  outcome  category — cost,  schedule, 
and  performance — the  overall  Likert  means  were  3.96,  3.84,  and  3.30,  respectively. 

One  hundred  and  sixty-eight  respondents  were  asked  to  rank  different  factors 
related  to  measuring  the  success  of  a  service  contract.  Of  the  168  respondents, 

46%  felt  that  process-related  factors  were  the  most  important.  Fifty-four  percent  felt 
that  outcome-related  factors  were  the  most  important.  The  distribution  of  highest 
ranked  responses  is  displayed  in  Figure  1 1 . 

Breaking  down  the  outcome-related  factors  further,  19%  of  respondents  felt 
that  cost-related  factors  were  the  most  important,  12%  felt  that  schedule-related 
factors  were  most  important,  and  23%  felt  that  performance-related  factors  were 
most  important.  All  results  for  aggregate  stakeholders  are  displayed  in  table  format 
in  Appendix  A,  Table  A2. 
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Figure  10.  Means  of  Aggregate  Stakeholder  Measurements  of  Success 
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Figure  1 1 .  Aggregate  Stakeholder  Ranking  of  Measurements  of  Success 

C.  Findings  by  Stakeholder 

While  examining  our  data,  we  found  that  we  obtained  the  most  responses 
from  certain  functional  roles.  PMs,  CORs,  and  COs/Contract  specialists  accounted 
for  87%  of  our  total  responses.  In  this  section,  we  examine  and  compare  how  each 
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of  these  functional  areas  defines  success  with  regards  to  service  contracts.  COs 
and  contract  specialists  were  grouped  under  one  category  and  from  now  on  the  term 
“CO”  encompasses  both  demographics.  We  differentiated  by  functional  role  and 
made  no  other  demographic  distinctions.  Our  data  include  functional  roles  across  all 
DAWIA  levels,  contract  types,  and  organizations. 

1.  PMs’,  CORs’,  and  COs’/Contract  Specialists’  Definitions  of 
Success 

a.  Program  Managers’  Definition  of  Success 

With  regard  to  the  Likert  scale  ratings  from  PMs,  we  found,  based  on  the  15 
responses  we  received,  that  this  group  considers  outcomes  slightly  more  important 
than  processes  when  defining  success.  When  defining  the  success  of  a  service 
contract,  PMs  responded  with  a  mean  rating  of  3.87  on  process-related  factors,  and 
a  mean  of  4.13  when  responding  to  questions  with  outcome-related  factors.  This  is 
a  small  range,  but  PMs  seem  to  feel  that  outcomes  are  somewhat  more  important 
when  defining  the  success  of  a  service  contract.  Our  results  are  displayed 
graphically  in  Figure  12,  with  outcomes  broken  out  with  respect  to  cost,  schedule, 
and  performance. 

Fifteen  respondents  were  asked  to  rank  different  factors  related  to  defining 
the  success  of  a  service  contract.  Of  the  1 5  respondents,  41  %  felt  that  process- 
related  factors  were  the  most  important.  Fifty-nine  percent  felt  that  outcome-related 
factors  were  the  most  important.  The  distribution  of  highest  ranked  responses  is 
displayed  in  Figure  13. 

Breaking  down  the  outcome-related  factors  further,  1 1  %  of  respondents  felt 
that  cost-related  factors  were  the  most  important,  22%  felt  that  schedule-related 
factors  were  most  important,  and  27%  felt  that  performance-related  factors  were 
most  important. 
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Figure  12.  PM  Definitions  of  Success 
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Figure  13.  PM  Ranking  of  Definitions  of  Success 

b.  Contracting  Officer  Representatives’  Definition  of  Success 

We  received  27  responses  from  CORs  to  Likert  scale-based  questions 
related  to  their  definitions  of  the  success  of  a  service  contract.  Process-related 
factors  received  a  mean  rating  of  3.97,  while  outcome-related  factors  produced  a 
mean  rating  of  4.15.  The  range  of  COR  definitions  on  the  Likert  scale  is  also 
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relatively  low,  and  CORs  also  appear  to  favor  outcomes  over  processes  when 
defining  the  success  of  a  service  contract.  Our  findings  are  displayed  graphically  in 
Figure  14. 

Twenty-seven  respondents  were  asked  to  rank  different  factors  related  to 
defining  the  success  of  a  service  contract.  Of  the  27  respondents,  39%  felt  that 
process-related  factors  were  the  most  important.  Sixty-one  percent  felt  that 
outcome-related  factors  were  the  most  important.  The  distribution  of  highest  ranked 
responses  is  displayed  in  Figure  15. 

Breaking  down  the  outcome-related  factors  further,  18%  of  respondents  felt 
that  cost-related  factors  were  the  most  important,  19%  felt  that  schedule-related 
factors  were  most  important,  and  24%  felt  that  performance-related  factors  were 
most  important. 
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Figure  14.  COR  Definitions  of  Success 
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COR  Ranking  of  Definitions  of  Success 


70% 


PROCESS  OUTCOME 


CATEGORIES 


■  PERFORMANCE 
SCHEDULE 

■  COST 

■  PROCESS 


Figure  15.  COR  Ranking  of  Definitions  of  Success 

c.  Contracting  Officers’  and  Contract  Specialists’  Definition  of 
Success 

We  received  responses  from  126  self-identified  COs/contract  specialists 
when  we  asked  them  to  rate  their  definitions  of  success  on  the  Likert  scale.  Our 
data  show  that  process-related  factors  achieved  an  overall  mean  rating  of  3.97,  and 
outcome-related  factors  received  an  overall  mean  rating  of  4.15.  These  means  are 
again  extremely  close  and  suggest  that  COs  only  slightly  favor  outcomes  versus 
processes  when  defining  the  success  of  a  service  contract.  Our  results  are 
displayed  graphically  in  Figure  16. 

On  the  Likert  scale,  COs/contract  specialists  rated  cost-related  factors  with  a 
mean  of  3.98,  they  rated  schedule-related  factors  with  a  mean  of  3.91,  and  they 
rated  performance-related  factors  with  a  mean  of  4.27. 

One  hundred  and  twenty-six  respondents  were  asked  to  rank  different  factors 
related  to  defining  the  success  of  a  service  contract.  Of  the  126  respondents,  40% 
felt  that  process-related  factors  were  the  most  important.  Sixty  percent  felt  that 
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outcome-related  factors  were  the  most  important.  The  distribution  of  highest  ranked 
responses  is  displayed  in  Figure  17. 

Breaking  down  the  outcome-related  factors  further,  17%  of  respondents  felt 
that  cost-related  factors  were  the  most  important,  15%  felt  that  schedule-related 
factors  were  most  important,  and  28%  felt  that  performance-related  factors  were 
most  important. 
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Figure  16.  CO  Definitions  of  Success 
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Figure  17.  CO  Ranking  of  Definitions  of  Success 

2.  PMs’,  CORs’,  COs’,  and  Contract  Specialists’  Measurement  of 
Success 

a.  Program  Managers’  Measurement  of  Success 

PMs  provided  15  responses  when  asked  to  respond  to  Likert  scale  questions 
related  to  how  they  measure  the  success  of  a  service  contract.  PMs’  ratings  of 
process-related  factors  received  an  overall  mean  of  2.52.  However,  outcomes 
produced  a  mean  of  3.78  on  the  Likert  scale.  This  shows  that  PMs  heavily  rely  on 
outcomes  rather  than  processes  to  measure  the  success  of  a  service  contract.  Our 
findings  are  displayed  graphically  in  Figure  18. 

On  the  Likert  scale,  cost-related  factors  showed  a  mean  of  4.15,  schedule- 
related  factors  received  a  mean  of  3.92,  and  performance-related  factors  received  a 
mean  of  3.25. 

Fifteen  respondents  were  asked  to  rank  different  factors  related  to  measuring 
the  success  of  a  service  contract.  Of  the  15  respondents,  43%  felt  that  process- 
related  factors  were  the  most  important.  Fifty-seven  percent  felt  that  outcome-related 
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factors  were  the  most  important.  The  distribution  of  highest  ranked  responses  is 
displayed  in  Figure  19. 

Breaking  down  the  outcome-related  factors  further,  1 3%  of  respondents  felt 
that  cost-related  factors  were  the  most  important,  14%  felt  that  schedule-related 
factors  were  most  important,  and  30%  felt  that  performance-related  factors  were 
most  important. 
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Figure  18.  PM  Measurements  of  Success 
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Figure  19.  PM  Ranking  of  Measurements  of  Success 

b.  Contracting  Officer  Representatives’  Measurement  of 
Success 

CORs  provided  27  responses  when  asked  to  respond  to  Likert  scale 
questions  related  to  how  they  measure  the  success  of  a  service  contract.  Process- 
related  factors  received  an  overall  mean  of  2.76  when  rated  by  this  functional  area. 
However,  outcomes  produced  a  mean  of  3.77  on  the  Likert  scale.  This  shows  that 
CORs  also  find  outcomes  significantly  more  important  than  processes  when 
measuring  the  success  of  a  service  contract.  Our  results  are  displayed  graphically 
in  Figure  20. 

Cost-related  factors  showed  a  mean  of  4.23,  schedule-related  factors  showed 
a  mean  of  4.08,  and  performance-related  factors  received  a  mean  of  2.99. 

Twenty-seven  respondents  were  asked  to  rank  different  factors  related  to 
measuring  the  success  of  a  service  contract.  Of  the  27  respondents,  39%  felt  that 
process-related  factors  were  the  most  important.  Sixty-one  percent  felt  that 
outcome-related  factors  were  the  most  important.  The  distribution  of  highest  ranked 
responses  is  displayed  in  Figure  21 . 
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Breaking  down  the  outcome-related  factors  further,  28%  of  respondents  felt 
that  cost-related  factors  were  the  most  important,  9%  felt  that  schedule-related 
factors  were  most  important,  and  24%  felt  that  performance-related  factors  were 
most  important. 
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Figure  20.  COR  Measurements  of  Success 
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Figure  21 .  COR  Ranking  of  Measurements  of  Success 


PtR  SCIfND/Ui 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-48  - 


c.  Contracting  Officers’  and  Contract  Specialists’ 
Measurement  of  Success 

COs/contract  specialists  provided  126  Likert  scale  responses  when  asked 
how  they  measure  the  success  of  a  service  contract.  Process-related  factors 
received  an  overall  mean  rating  of  2.41  from  COs/contract  specialists.  Conversely, 
outcome-related  factors  were  rated  with  a  mean  of  3.77.  Our  results  are  displayed 
graphically  in  Figure  22. 

Within  the  outcome-related  factors,  cost  showed  a  mean  of  3.98,  schedule 
received  a  mean  rating  of  3.98,  and  performance  a  mean  of  3.47.  This  shows  that 
COs  and  contract  specialists  are  aligned  with  PMs  and  CORs  when  measuring  the 
success  of  a  service  contract.  All  three  functional  roles  agree  that  outcome-related 
factors  heavily  outweigh  process-related  factors  when  measuring  success. 

One  hundred  and  twenty-six  respondents  were  asked  to  rank  different  factors 
related  to  measuring  the  success  of  a  service  contract.  Of  the  126  respondents, 
49%  felt  that  process-related  factors  were  the  most  important.  Fifty-one  percent  felt 
that  outcome-related  factors  were  the  most  important.  The  distribution  of  highest 
ranked  responses  is  displayed  in  Figure  23. 

Breaking  down  the  outcome-related  factors  further,  17%  felt  that  cost-related 
factors  were  the  most  important,  11%  felt  that  schedule-related  factors  were  most 
important,  and  23%  felt  that  performance-related  factors  were  most  important.  All 
results  by  individual  stakeholders  are  displayed  in  table  format  in  Appendix  A,  Table 
A4. 
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Figure  22.  CO  Measurements  of  Success 
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Figure  23.  CO  Ranking  of  Measurements  of  Success 

D.  Findings  by  DAWIA  Level 

We  separated  our  data  on  DAWIA  certification  levels  in  order  to  examine  and 
analyze  the  results.  This  analysis  is  strictly  divided  among  DAWIA  levels  and 
contains  no  other  demographic  differentiation.  Of  the  168  participants,  85%  had  a 
DAWIA  certification  level  of  I,  II,  or  III.  The  education  and  experience  requirements 
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for  certification  rise  in  conjunction  with  the  level.  A  majority  of  data  collected  on 
Level  I  participants  showed  that  these  participants  had  fewer  than  three  years  of 
experience  in  an  acquisition-related  billet.  Level  II  participants  had  a  more  diverse 
range  of  experience  with  the  majority  having  more  than  four  years,  but  fewer  than 
eight  years  of  experience  in  an  acquisition-related  billet.  Level  III  participants  also 
showed  a  diverse  range  of  experience,  and,  as  expected,  contained  participants  with 
more  than  19  years  of  experience  in  an  acquisition-related  billet. 

1.  DAWIA-Certified  Participants’  Definition  of  Success 

When  differentiated  by  DAWIA  level,  Level  I  and  II  respondents  showed  a 
slightly  higher  rating  than  Level  III  participants  on  Likert  scale  factor-related 
questions.  However,  the  responses  concerning  processes  versus  outcomes  showed 
little  difference  within  each  level.  Level  I  participants  displayed  a  mean  rating  on 
process  and  outcome  factors  of  4.00  and  4.14,  respectively.  Level  II  participants 
displayed  a  mean  rating  on  process  and  outcome  factors  of  4.16  and  4.17, 
respectively.  Level  III  participants  displayed  a  mean  rating  on  process  and  outcome 
factors  of  3.79  and  4.01 ,  respectively.  This  shows  that  while  distinct  levels  may  rate 
processes  and  outcomes  differently,  they  feel  that  outcomes  are  slightly  more 
important  than  processes.  Yet,  processes  are  still  considered  important  when 
defining  the  success  of  service  contracts.  Our  results  are  displayed  graphically 
across  all  certification  levels  in  Figure  24. 

One  hundred  and  forty-two  respondents  with  a  DAWIA  certification  provided 
responses  when  asked  to  rank  definitions  of  success  concerning  a  service  contract. 
Each  certification  level  provided  similar  answers  within  each  of  the  categories  on 
processes  and  outcomes.  Of  the  Level  I  respondents,  42%  felt  that  process-related 
factors  were  the  most  important,  while  58%  felt  that  outcome-related  factors  were 
the  most  important.  Of  the  Level  II  respondents,  39%  felt  that  process-related 
factors  were  the  most  important,  while  61%  felt  that  outcome-related  factors  were 
the  most  important.  Of  the  Level  III  respondents,  40%  felt  that  process-related 
factors  were  the  most  important,  while  60%  felt  that  outcome-related  factors  were 
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the  most  important.  Again,  this  shows  that  these  particular  demographics  tend  to 
use  outcomes  when  defining  the  success  of  a  service  contract.  The  distribution  of 
the  responses  is  shown  in  Figure  25. 
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Figure  24.  Definitions  of  Success  Across  DAWIA  Levels 
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Figure  25.  Ranking  of  Definitions  of  Success  Across  DAWIA  Levels 
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2.  DAWIA-Certified  Participants’  Measurement  of  Success 

Our  data  suggest  that  when  measuring  the  success  of  a  service  contract,  the 
various  DAWIA  levels  rely  heavily  on  outcome-related  factors  rather  than  process- 
related  factors.  Level  I  participants  displayed  a  mean  rating  on  process  and 
outcome  factors  of  2.20  and  3.76,  respectively.  Level  II  participants  displayed  a 
mean  rating  on  process  and  outcome  factors  of  2.63  and  3.86,  respectively.  Level  III 
participants  displayed  a  mean  rating  on  process  and  outcome  factors  of  2.22  and 
3.72,  respectively.  Our  results  are  displayed  graphically  across  all  certification  levels 
in  Figure  26. 

One  hundred  and  forty-six  respondents  with  a  DAWIA  certification  provided 
146  responses  when  asked  to  rank  measurement  factors  of  success  concerning  a 
service  contract.  There  is  a  little  more  diversity  between  the  levels,  but  all  favor 
outcomes  over  processes.  Forty-nine  percent  of  Level  I  respondents  felt  that 
processes  were  most  important,  with  51%  favoring  outcomes.  Forty-eight  percent  of 
Level  II  respondents  felt  that  processes  were  most  important,  with  52%  favoring 
outcomes.  Forty-three  percent  of  Level  III  respondents  felt  that  processes  were 
most  important,  with  57%  favoring  outcomes.  The  distribution  of  the  responses  is 
shown  in  Figure  27.  All  results  by  DAWIA  level  are  displayed  in  table  format  in 
Appendix  A,  Table  A6. 
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Figure  26.  Measurement  of  Success  Across  DAWIA  Levels 
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Figure  27.  Ranking  of  Measurements  of  Success  Across  DAWIA  Levels 

E.  Data  Analysis  by  Type  of  Service 

While  examining  our  data,  we  found  that  we  obtained  the  majority  of 
responses  from  participants  who  procured  certain  types  of  services.  Of  the  168  total 
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responses,  the  equipment-related  service  portfolio  grouping  accounted  for  72 
responses  and  the  knowledge-based  service  portfolio  groupings  for  77  responses. 
Given  the  majority  of  responses  from  these  two  groups,  we  further  analyzed  them  to 
identify  trends  across  service  types.  The  equipment-related  service  portfolio 
grouping  includes  the  procurement  of  maintenance,  repair  and  overhaul,  equipment 
modification,  installation,  and  quality  control  services.  The  knowledge-based 
services  portfolio  grouping  is  composed  of  professional  and  administrative  services, 
engineering  management,  program  management,  logistics  management,  and 
education  and  training.  Further  demographic  breakdown  of  the  two  groups  shows 
40%  of  knowledge-based  service  participants  were  involved  with  cost 
reimbursement-type  contracts  compared  to  only  6%  for  equipment-related  services. 
The  higher  percentage  of  cost  reimbursement-type  contracts  for  knowledge-based 
service  participants  is  most  likely  due  to  the  increased  challenges  and  uncertainties 
in  defining  requirements  associated  with  these  types  of  services.  Equipment-related 
services  are  generally  more  concrete  in  terms  of  requirements  definitions  and  would 
be  more  suitable  for  a  fixed-price  contractual  instrument. 

In  this  section,  we  examine  and  compare  how  stakeholders  involved  with 
equipment-related  services  and  knowledge-based  services  define  and  measure 
success  with  regards  to  services  contracts.  We  only  differentiated  by  type  of  service 
and  made  no  other  demographic  distinctions  for  mean  and  ranking  results.  Our  data 
include  the  two  predominant  types  of  services  procured  across  all  functional  roles, 
DAWIA  levels,  contract  types,  and  organizations. 

1.  Specific  Service  Type  Definition  of  Success 

In  response  to  our  questions  asking  participants  to  classify  different  factors 
related  to  defining  the  success  of  a  service  contract,  we  received  149  responses 
from  participants  who  work  on  knowledge-based  and  equipment-related  services. 

We  received  72  responses  from  participants  who  worked  on  equipment-related 
services  and  77  responses  from  those  involved  with  knowledge-based  services. 
When  differentiating  between  types  of  service,  we  found  that  equipment-related 
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service  participants  rated  both  processes  and  outcomes  higher  on  all  Likert  scale 
questions  than  knowledge-based  service  participants.  Equipment-related 
participants  displayed  a  mean  rating  on  process  and  outcome  factors  of  4.05  and 
4.10,  respectively.  Knowledge-based  participants  displayed  a  mean  rating  on 
process  and  outcome  factors  of  3.92  and  4.02,  respectively.  This  shows  that 
participants  segregated  by  type  of  service  may  rate  processes  and  outcomes  slightly 
differently;  however,  they  both  indicated  that  outcomes  are  slightly  more  important 
than  processes  for  defining  success  on  Likert  scale  questions.  Our  results  are 
displayed  graphically  by  type  of  service  in  Figure  28. 

Seventy-two  respondents  involved  with  equipment-related  services  provided 
responses  when  asked  to  rank  definitions  of  success  concerning  a  service  contract. 
The  questions  asked  of  participants  dealt  with  different  aspects  that  aligned  with 
process-  and  outcome-related  groupings.  Of  the  72  respondents,  42%  believed  that 
process  factors  are  most  important,  while  58%  felt  that  outcomes  more  accurately 
define  the  success  of  a  service  contract. 

Breaking  down  the  outcome-related  factors  further,  15%  felt  that  cost-related 
factors  were  the  most  important,  18%  felt  that  schedule-related  factors  were  most 
important,  and  25%  felt  that  performance-related  factors  were  most  important. 

Seventy-seven  respondents  involved  with  knowledge-based  services 
provided  responses  when  asked  to  rank  definitions  of  success  concerning  a  service 
contract.  Of  the  77  respondents,  39%  believed  that  process  factors  are  most 
important,  while  61%  felt  that  outcomes  more  accurately  defined  the  success  of  a 
service  contract.  The  results  show  that  equipment-related  service  participants  rated 
processes  higher  and  outcomes  lower  than  knowledge-based  participants,  but  both 
groups  indicated  that  outcomes  are  more  important  than  processes  for  defining 
success,  based  on  their  responses  to  ranking  questions.  The  distribution  of  highest 
ranked  responses  is  displayed  in  Figure  29. 
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Breaking  down  the  outcome-related  factors  further,  1 6%  of  respondents  felt 
that  cost-related  factors  were  the  most  important,  18%  felt  that  schedule-related 
factors  were  most  important,  and  28%  felt  that  performance-related  factors  were 
most  important. 
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Figure  28.  Definitions  of  Success  by  Service  Type 
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Figure  29.  Ranking  of  Definitions  of  Success  by  Service  Type 
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2.  Specific  Service  Type  Measurements  of  Success 

In  measuring  the  success  for  different  types  of  services,  the  equipment- 
related  service  participants  rated  processes  much  higher  on  Likert  scale  questions 
than  knowledge-based  service  participants.  Equipment-related  service  participants 
also  rated  outcomes  as  being  of  lower  importance  than  did  knowledge-based  service 
participants.  Equipment-related  participants  displayed  a  mean  rating  on  process 
and  outcome  factors  of  2.68  and  3.76,  respectively.  Knowledge-based  participants 
displayed  a  mean  rating  on  process  and  outcome  factors  of  2.29  and  3.82, 
respectively.  The  results  show  that  equipment-related  service  participants  rated 
processes  higher  and  outcomes  lower  than  knowledge-based  participants,  but  both 
groups  indicated  that  outcomes  are  more  important  than  processes  for  measuring 
success,  based  on  their  responses  to  Likert  scale  questions.  Our  results  are 
displayed  graphically  by  type  of  service  in  Figure  30. 

In  response  to  our  questions  asking  participants  to  classify  different  factors 
related  to  measuring  the  success  of  a  service  contract,  we  received  149  responses 
from  participants  who  work  on  knowledge-based  and  equipment-related  services. 
The  questions  dealt  with  process-  and  outcome-related  groupings.  Of  the  72 
equipment-related  responses  to  our  ranking  questions,  process-related  factors  were 
ranked  most  important  44%  of  the  time,  while  outcome-related  factors  were  ranked 
as  most  important  56%  of  the  time. 

Breaking  down  the  outcome-related  factors  further,  21%  of  respondents  felt 
that  cost-related  factors  were  the  most  important,  12%  felt  that  schedule-related 
factors  were  most  important,  and  21%  felt  that  performance-related  factors  were 
most  important. 

Forty-eight  percent  of  the  77  respondents  associated  with  knowledge-based 
services  felt  that  process-related  factors  were  the  most  important  when  measuring 
success.  Fifty-two  percent  of  respondents  felt  that  outcome-related  factors  were  the 
most  important.  The  results  show  that  equipment-related  service  participants  rated 
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processes  lower  and  outcomes  higher  than  knowledge-based  participants,  but  both 
groups  indicated  that  outcomes  are  more  important  than  processes  for  measuring 
success,  based  on  their  responses  to  our  ranking  questions.  The  distribution  of 
highest  ranked  responses  is  displayed  in  Figure  31 . 

Breaking  down  the  outcome-related  factors  further,  1 6%  of  respondents  felt 
that  cost-related  factors  were  the  most  important,  1 1%  felt  that  schedule-related 
factors  were  most  important,  and  25%  felt  that  performance-related  factors  were 
most  important.  All  results  by  service  type  are  displayed  in  table  format  in  Appendix 
A,  Table  A8. 
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Figure  30.  Measurement  of  Success  by  Service  Type 
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Figure  31 .  Ranking  of  Measurement  of  Success  by  Service  Type 
F.  Analysis 

1 .  Analysis  of  Aggregate  Survey  Responses 

The  research  findings  from  our  overall  survey  show  that  when  asked  to 
respond  on  a  Likert  scale,  different  stakeholders  find  all  aspects  of  processes  and 
outcomes  important  when  defining  the  success  of  a  service  contract.  The  means  of 
the  responses  we  collected  are  very  close,  and  it  does  not  seem  that,  as  a  whole, 
our  population  favors  process  or  outcome  when  defining  success.  This  is  due  to  the 
nature  of  Likert  scale  questions.  When  asked  if  something  such  as  cost  overruns, 
major  milestones,  or  a  lack  of  protests  is  important,  all  stakeholders  will  invariably 
say  “yes.”  That  is  why  the  overall  mean  of  all  responses,  for  both  outcomes  and 
processes,  is  fairly  high  at  4.03.  When  respondents  were  forced  to  rank,  the 
responses  differed  and  outcome-related  responses  received  a  high  rank  of  1  or  2 
60%  of  the  time.  This  is  because  outcomes,  such  as  keeping  on  schedule  and 
adhering  to  a  budget,  are  easy  to  define  and  understand.  Process-related  factors 
such  as  administration  and  communication  are  harder  to  quantify. 
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The  findings  also  demonstrate  that  when  measuring  the  success  of  a  service 
contract,  all  stakeholders  tend  to  focus  on  outcomes  and  do  not  take  into 
consideration  the  processes;  this  was  true  for  both  Likert  scale  responses  and 
ranking  responses.  This  is  very  evident  in  the  Likert  scale  responses,  where  none  of 
the  process-related  factors  showed  a  mean  of  3  or  more.  When  forced  to  rank  the 
different  factors  with  respect  to  measuring  success,  the  results  were  similar  to 
defining  success,  with  56%  of  “most  important”  responses  falling  under  the 
outcomes  category. 

In  general  our  findings  from  the  “other”  category  mirrored  our  aggregate 
results.  While  there  were  only  10  responses,  all  respondents  felt  that  outcomes 
were  the  most  important  factor  when  defining  and  measuring  the  success  of  a 
service  contract.  We  found  that  our  stakeholders  in  this  category  rated  and  ranked 
processes  extremely  low  in  both  defining  and  measuring  the  success  of  a  service 
contract.  This  is  because  these  stakeholders  are  not  terribly  burdened  by 
administration  and  other  process-related  factors  so  they  feel  that  these  factors  are 
not  important.  For  example,  a  contractor  or  end  user  does  not  necessarily  conduct 
market  research  or  choose  the  appropriate  contract  type.  However,  they  are  very 
concerned  with  staying  within  cost,  keeping  up  with  schedule,  and  maintaining  a 
high  level  of  performance. 

2.  Analysis  Across  Stakeholders 

When  examining  how  different  stakeholders  define  the  success  of  a  service 
contract,  we  found  that  PMs,  CORs,  COs,  and  contract  specialists  all  agree  that 
outcome  is  slightly  more  important  than  process,  based  on  their  ratings  of  separate 
factors  on  a  Likert  scale.  Each  functional  role  rated  outcome  slightly  over  4.00,  while 
rating  processes  just  below  4.00.  The  mean  of  the  functional  roles  combined  was 
3.94  for  processes  and  4.1 1  for  outcomes.  Within  outcome,  performance-related 
factors  received  the  highest  average  rating,  while  schedule-related  factors  received 
the  lowest  average  rating.  All  functional  roles  showed  an  upward  trend  from 
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schedule,  to  cost,  to  performance.  A  comparison  of  our  Likert  scale  findings  for 
defining  success  across  functional  roles  is  displayed  graphically  in  Figure  32. 

When  stakeholders  were  asked  to  rank  different  factors  concerning  their 
definition  of  success,  we  found  that  there  was  clear  agreement  that  outcomes  are 
more  important  than  processes.  There  is,  however,  some  disagreement  within  the 
outcome  factors  of  cost,  schedule,  and  performance.  CORs  feel  that  cost  is  the  most 
important  factor,  while  PMs,  COs,  and  specialists  placed  performance  at  the  top  of 
their  rankings.  Examined  collectively,  the  major  stakeholders  provided  168 
responses  when  ranking  their  definition  of  the  success  of  a  service  contract.  Sixty 
percent  of  respondents  felt  that  outcome-related  factors  were  most  important,  while 
40%  felt  that  process-related  factors  were  the  most  important  when  defining 
success.  The  distribution  of  highest  ranked  responses  is  displayed  in  Figure  33. 
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Figure  32.  Definitions  of  Success  Across  Major  Stakeholders 
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Figure  33.  Major  Stakeholder  Ranking  of  Definitions  of  Success 

According  to  our  data,  stakeholders  also  tend  to  measure  success  in  the 
same  way.  When  asked  to  rate  different  factors  on  the  Likert  scale  related  to 
stakeholders’  measures  of  success,  all  respondents  agreed  that  outcomes  far 
outweigh  processes.  When  looking  at  the  mean  across  stakeholders,  processes 
received  a  rating  of  2.56,  while  outcomes  received  a  rating  of  3.78.  Within  outcome- 
related  factors,  stakeholders  showed  an  upward  trend  from  performance,  to 
schedule,  to  cost.  A  comparison  of  our  findings  for  defining  success  on  the  Likert 
scale  across  functional  roles  is  displayed  graphically  in  Figure  34. 

Our  ranking  data  show  that,  again,  major  stakeholders  prefer  outcome-related 
factors  when  measuring  the  success  of  service  contracts.  When  examined  in 
aggregate,  the  major  stakeholders  provided  168  responses  to  our  ranking  questions. 
Of  these  responses,  43%  of  respondents  felt  process  factors  were  most  important, 
while  57%  favored  factors  related  to  outcomes.  The  distribution  of  highest  ranked 
responses  is  displayed  in  Figure  35. 
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Figure  34.  Measurement  of  Success  Across  Major  Stakeholders 


_  80% 

~r  60% 

z 

£  40% 
O  20% 

Q_ 

cn 

£  0% 


Major  Stakeholder  Ranking  of 
Measurements  of  Success 


PROCESS  OUTCOME 
COR  COR 


■ 


PROCESS  OUTCOME 
PCO  PCO 

CATEGORIES 


PROCESS  OUTCOME 
PM  PM 


PROCESS 


COST 


SCHEDULE 


PERFORMANCE 


Figure  35.  Major  Stakeholder  Ranking  of  Measurements  of  Success 

The  data  collected  from  the  major  stakeholders  are  similar  to  our  cumulative 
findings.  That  is  to  be  expected,  considering  they  made  up  87%  of  our  total 
population.  The  Likert  scale  responses  for  definitions  of  success  were,  again, 
relatively  high  and  this  is  due  to  the  reason  explained  earlier.  It  is  interesting  that  in 
both  defining  and  measuring  success,  CORs  ranked  cost  highest  out  of  the  three 
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stakeholders.  In  previous  research  (Miller  et  al.,  2012),  CORs  listed  performance  as 
more  important  than  cost  when  defining  and  measuring  success.  This  is  probably 
due  to  the  open-ended  nature  of  the  questions  asked  in  previous  research.  Our 
survey  may  have  brought  to  light  issues  or  factors  that  CORs  had  never  thought  of 
before. 

Another  interesting  result  is  that  COs  tend  to  place  nearly  equal  importance 
on  processes  and  outcomes  when  forced  to  rank  factors  concerning  measuring 
success.  This  is  probably  due  to  the  administrative  nature  of  the  COs’  role.  For 
example,  their  functional  role  has  to  deal  with  modifications,  COR  reports,  and 
exercising  options.  PMs  and  CORs  are  not  overly  concerned  with  processes  and 
are  focused  on  the  requirement  and  outcomes.  The  data  reflect  this  fact. 

It  is  of  note  that  every  demographic  consistently  rated  processes  significantly 
higher  on  the  Likert  scale  when  defining  success  versus  measuring  success.  We 
feel  that  this  is  because  stakeholders  view  measures  as  a  tangible  entity  associated 
with  post-award  functions.  Measures  such  as  cost,  schedule,  and  performance  are 
fairly  straightforward  in  as  much  as  either  a  goal  is  met  or  it  is  not.  Processes  such 
as  communication  flow  and  overall  management  are  more  obscure  and  subjective. 
The  stakeholders  rated  processes  higher  for  defining  success  because  they  are 
closely  associated  with  mainly  pre-award  functions.  Processes  such  as  choosing 
the  correct  contract  type  and  appropriate  proposal  evaluation  are  crucial  for 
success.  Because  these  are  pre-award  activities,  it  is  easier  to  define  success  than 
to  measure  it. 


We  performed  a  statistical  analysis  across  the  major  stakeholders  to 
determine  if  there  was  a  significant  difference  between  the  ratings  on  the  Likert 
scale.  We  first  performed  an  F-test  two  sample  for  variances  to  determine  the 
appropriate  f-test  to  perform.  In  all  instances,  we  found  an  equal  variance  among 
stakeholders.  The  only  significant  difference  we  discovered  was  between  the  CORs 
and  COs/specialists  when  measuring  success.  This  could  be  due  to  the  fact  that 
CORs  view  communication  and  other  processes  as  a  key  factor  when  measuring  the 
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success  of  a  service  contract.  CORs  are  also  likely  to  view  a  protest  as  a  serious 
issue  when  measuring  success  because  it  results  in  a  delay  of  execution  and  the 
CORs  cannot  perform  their  duties.  There  was  no  statistically  significant  difference 
between  any  other  of  the  stakeholders  on  the  Likert  scale.  Tables  from  statistical 
analysis  are  located  in  Appendix  B. 

3.  Analysis  Across  DAWIA  Levels 

After  further  assessment,  we  noted  that  a  majority  of  the  DAWIA  level- 
certified  personnel  were  mostly  COs,  so  their  results  somewhat  mirror  that 
demographic.  It  is  of  note  that  when  we  analyzed  Likert  scale  responses,  Level  III 
personnel  tend  to  provide  lower  responses  on  the  scale  than  Levels  I  and  II  when 
defining  success.  Level  III  personnel  generally  have  more  experience  than  Levels  I 
and  II,  and  that  may  be  the  reason  for  this  trend.  Seventy-two  percent  of  Level  Ill- 
certified  respondents  reported  having  10  or  more  years  of  experience,  and  this 
demographic  also  had  the  most  divergence  from  the  CO  role.  The  level  of 
experience  can  explain  the  low  Likert  score  for  the  process  factors  for  participants 
with  a  Level  III  certification.  Because  those  respondents  have  10  or  more  years  of 
experience,  the  processes  have  become  routine  and  they  view  administration  and 
communication  as  standard,  whereas  Level  I  and  II  respondents  have  relatively  low 
experience  and  believe  that  processes  are  more  important.  Another  reason  that 
more  experienced  stakeholders  view  processes  as  less  important  is  because,  as  a 
result  of  their  experience,  they  have  seen  the  outcomes  of  fully  completed  contracts 
whereas  more  junior  personnel  may  have  only  dealt  with  award  and  administration. 
There  is  the  possibility  that  they  have  never  conducted  close  out  on  a  contract  to 
actually  see  the  true  outcome,  and,  as  a  result,  Level  I  and  II  stakeholders  believe 
that  processes  are  more  important  than  do  Level  III  stakeholders. 


4.  Analysis  by  Type  of  Service 

The  findings  based  on  type  of  service  showed  no  substantial  deviation  for 
defining  the  success  of  a  services  contract.  Both  equipment-related  and  knowledge- 
based  groups  rated  outcomes  slightly  higher  than  processes,  based  on  their 
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responses  to  the  Likert  scale  questions.  Ranking  questions  for  definitions  of 
success  showed  outcomes  as  more  important  than  processes  for  both  groups  and 
corroborated  the  Likert  scale  mean  findings.  Both  Likert  scale  and  ranking  question 
results  showed  performance  to  be  the  most  important  component  of  outcomes.  For 
example,  performance  results  for  the  two  groups  had  means  of  4.28  and  4.24, 
compared  to  3.97  and  3.82  for  schedule.  Ranking  questions  showed  25%-28%  for 
performance  and  18%  for  schedule  across  both  groups.  Perhaps  ranking  results, 
due  to  the  limitation  of  choices,  provide  the  most  precise  definitions  of  success. 
When  participants  were  forced  to  rank,  they  emphasized  the  importance  of 
outcomes  (cost,  schedule,  and  performance)  over  processes  more  definitively  when 
defining  the  characteristics  of  successful  service  contracts. 

The  findings  for  measuring  success  also  showed  no  substantial  deviation 
based  on  type  of  service.  Both  equipment-related  and  knowledge-based  groups 
rated  outcomes  significantly  higher  than  processes  on  the  Likert  scale  questions. 
Ranking  questions  for  measuring  success  showed  outcomes  as  more  important  than 
processes  for  both  groups  and  corroborated  the  Likert  scale  mean  findings. 
Participants  who  work  on  equipment-related  services  rated  cost  as  more  important 
and  performance  as  less  important  on  both  Likert  and  ranking  questions  compared 
to  knowledge-based  participants.  This  finding  seems  to  indicate  that  for  knowledge- 
based  services,  stakeholders  were  more  willing  to  trade  cost  for  enhanced 
performance.  This  may  be  attributable  to  the  highly  specialized  and  technical  nature 
of  functions  such  as  engineering  management  and  program  management,  which  the 
government  lacks  the  crucial  internal  capacity  to  perform. 


G.  Summary 

In  this  chapter,  we  presented  and  analyzed  the  various  responses  that  we 
received  to  our  survey  questions.  First,  we  looked  at  responses  from  all 
demographics  and  determined  that  all  stakeholders  believe  processes  and 
outcomes  are  important  when  defining  the  success  of  a  service  contract.  Our  results 
also  showed  that  when  measuring  success,  all  demographics  placed  higher 
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importance  on  outcomes  rather  than  processes.  We  then  differentiated  the 
demographics  based  on  the  highest  percentage  of  responses.  These  demographics 
included  the  major  stakeholders,  DAWIA  certification  level,  and  type  of  service 
procured.  In  all  instances,  we  found  that  outcomes  outweighed  processes  when 
defining  and  measuring  the  success  of  a  service  contract. 
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V. 


Conclusions,  Recommendations,  and  Areas 
for  Further  Research 


A.  Conclusions 

Over  the  last  10  years,  the  DoD  has  steadily  shown  increases  in  the  number 
of  dollars  spent  on  the  acquisition  of  services,  accounting  for  over  half  of  total 
contract  spending.  This  increased  reliance  on  service  contractors  has  prompted  the 
GAO  and  DoDIG  to  increase  surveillance  and  issue  numerous  reports  citing 
deficiencies  and  weaknesses  in  the  acquisition  and  contract  management  process. 
These  flaws  were  noted  in  every  aspect  of  the  contracting  process,  such  as 
insufficient  market  research,  choosing  an  inappropriate  contract  type,  poor 
contractor  oversight,  and  lack  of  properly  trained  government  personnel. 

Another  major  issue  is  that  while  these  spending  increases  have  occurred, 
the  acquisition  workforce  has  steadily  decreased.  The  knowledge  gap  created  from 
this  decrease  has  contributed  significantly  to  ineffective  contract  management  over 
the  last  decade.  Well-trained  and  capable  personnel  are  essential  for  successful 
management  of  service  contracts.  It  is  imperative  that  the  correct  metrics  are 
utilized  to  define  and  measure  that  success. 

This  research  project  was  conducted  to  further  analyze  services  contracting 
management  practices  within  the  Navy.  The  goal  was  to  answer  three  research 
questions  in  order  to  strengthen  the  Navy’s  understanding  of  successful  services 
acquisitions  from  disparate  stakeholder  perspectives: 

How  do  different  stakeholders  define  successful  services  contracts 
within  the  Navy? 

How  do  different  stakeholders  measure  services  contracts  within  the 
Navy? 

How  should  Navy  services  contracts  be  defined  and  measured? 
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The  objective  of  this  research  was  to  build  upon  the  understanding  developed 
in  prior  research  projects  to  generate  metrics  for  defining  and  measuring  successful 
services  contracts.  These  developed  metrics  could  help  identify  factors  that 
influence  successful  services  contract  outcomes. 

We  found  that  when  defining  success  all  stakeholders  tend  to  utilize  outcome- 
related  factors  over  process-oriented  factors.  This  is  because  outcomes  tend  to 
drive  perceptions  of  success  more  than  processes.  Outcomes  such  as  customer 
satisfaction  or  meeting  major  milestones  are  easily  identified,  whereas  processes 
tend  to  be  more  ambiguous.  A  process,  such  as  conducting  market  research  in 
accordance  with  the  FAR,  does  not  necessarily  define  success.  Based  on  our 
findings,  processes  are  important  factors  in  that  they  contribute  to  the  overall 
success,  but  not  necessarily  the  definition  success.  For  example,  public  policy 
dictates  that  the  government  shall  provide  for  full  and  open  competition,  award  to 
small  business  when  appropriate,  and  negotiate  fair  and  reasonable  prices.  These 
are  all  statutory  requirements.  Based  on  our  research  findings,  these  processes  do 
not  necessarily  define  the  success  of  a  service  contract;  however,  they  will 
determine  if  a  contract  was  awarded  in  accordance  with  federal  law  and  public 
policy.  Thus,  it  is  conceivable  that  a  contract  may  be  defined  as  successful  in  terms 
of  outcomes  (cost,  schedule,  performance)  but,  based  on  processes,  may  be  in 
violation  of  public  policy. 

The  findings  with  relation  to  measuring  the  success  of  a  service  contract  were 
comparable  to  defining  success.  More  participants  favored  outcomes,  but  this  can 
also  be  explained  by  looking  at  the  factors  within  the  broad  category.  Cost  and 
schedule  are  easy  factors  to  track  and  the  results  can  generally  be  quantified.  If  a 
schedule  is  not  met,  then  the  contract  could  be  considered  unsuccessful.  However, 
when  there  is  no  appropriate  contractor  oversight,  a  process,  success  is  not 
necessarily  sacrificed. 

We  found  that  stakeholders  are  inclined  to  define  and  measure  success  in 
similar  ways  when  asked  about  differing  degrees  of  importance,  agreement,  and 
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likely  occurrence  of  events  throughout  the  service  contracting  cycle.  When  asked  to 
rate  the  factors  on  a  Likert  scale,  all  stakeholders  agree  that  process-related  factors 
such  as  communication,  administrative  burden,  and  clear  objectives  are  of  high 
importance.  They  also  agree  that  outcomes  relating  specifically  to  cost,  schedule, 
and  performance  are  important  when  defining  the  success  of  a  contract.  However, 
when  asked  to  rank  these  factors,  stakeholders  favor  outcomes  rather  than 
processes  when  defining  success. 

We  discovered  that  when  measuring  the  success  of  a  service  contract  all 
stakeholders  feel  that  outcomes  are  better  measures  of  success  than  processes,  as 
indicated  by  their  responses  both  on  the  Likert  scale  and  when  ranking  separate 
factors.  This  was  revealed  because  outcomes  are  easily  quantifiable.  It  is  fairly 
easy  to  track  costs,  rate  performance,  and  differentiate  between  a  contract  that  is  on 
schedule  and  one  that  is  falling  behind  schedule.  It  is  not  as  easy  to  determine  an 
appropriate  level  of  administrative  work,  a  proper  amount  of  communication  between 
stakeholders,  or  the  correct  number  of  change  orders  or  modifications  that  are 
necessary. 

We  also  found  that  CORs  view  processes  as  more  important  than 
COs/specialists  when  measuring  the  success  of  a  service  contract.  CORs,  ideally, 
are  in  frequent  communication  with  the  COs/specialists  to  convey  the  progress  and 
performance  on  a  contract  so  the  CORs  view  processes  as  significantly  more 
important.  Another  reason  that  CORs  rate  processes  more  favorably  than 
COs/specialists  is  that  protests  will  completely  stop  work  on  a  contract,  and  the 
CORs  believe  that  a  protest  will  result  in  an  unsuccessful  contract. 


Our  research  shows  that  the  metrics  used  to  measure  success  are  typically 
related  to  cost,  schedule,  and  performance.  There  is,  however,  no  general 
consensus  among  stakeholders  as  to  what  can  be  interpreted  as  the  most  important 
of  these  metrics.  According  to  the  data  we  collected,  some  demographics 
considered  performance  the  most  important  measure,  whereas  others  found  cost  to 
be  the  most  important  measure  of  success. 
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B.  Recommendations 


Process-related  factors  need  to  be  emphasized  in  stakeholders’  definitions  of 
success.  Outcome-related  factors  of  cost,  schedule,  and  performance  are  dominant, 
while  processes  are  viewed  with  little  importance  when  defining  success.  It  is 
evident  that  this  is  an  issue.  Choosing  the  appropriate  contract  type,  a  step  in  the 
planning  process,  should  be  of  great  importance  when  defining  the  success  of  a 
service  contract.  Time-and-materials  or  cost-type  contracts  could  result  in 
overpayment,  and  cost  is  considered  an  important  factor  when  defining  success.  A 
well-written  and  clear  statement  of  work  should  also  be  highlighted  when  defining 
success.  If  the  contractor  or  the  acquisition  team  does  not  understand  the 
requirement,  modifications  are  necessary,  and  increased  costs  or  a  schedule  slip 
could  result.  Process-related  factors  are  inextricably  tied  to  the  definition  of  success 
of  a  service  contract,  and  steps  need  to  be  taken  so  that  stakeholders  understand 
and  act  on  their  importance. 

Internal  control  measures  should  be  in  place  to  ensure  that  proper  processes 
are  being  followed  and  that  all  stakeholders  place  high  importance  on  the  value  of 
these  processes.  If  stakeholders  are  forced  to  take  into  consideration  processes 
when  measuring  and  defining  the  success  of  a  service  contract,  some  of  the 
deficiencies  in  services  procurement  could  be  corrected.  An  example  of  an  internal 
control  measure  is  the  Services  Requirements  Review  Board,  a  program  that 
NAVSEA  implemented  in  2011.  Commonly  referred  to  as  “Services  Court,”  the 
program  is  an  annual  review  of  the  full  range  of  NAVSEA  service  requirements  to 
understand  what  services  are  required  and  ensure  proper  oversight  is  in  place  so 
that  there  is  maximum  value  for  dollars  spent  (NAVSEA,  2012).  The  board  is 
chaired  by  the  NAVSEA  Commander,  Vice  Admiral  Kevin  McCoy,  and  Executive 
Director  Brian  Persons  in  order  to  involve  leadership  at  the  very  top  level.  As  a 
result  of  their  findings,  NAVSEA  has  modified  their  policies  to  ensure  a  clearer 
definition  of  service  requirements,  require  more  stringent  contract  file  maintenance, 
and  implement  proper  surveillance  and  proper  performance  reporting  requirements 
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(NAVSEA,  2012).  All  of  these  reforms  are  process  related  and  other  commands 
could  benefit  by  implementing  similar  internal  control  measures. 


Another  way  to  increase  the  emphasis  placed  on  processes  when  defining 
and  measuring  success  is  to  put  in  place  an  operational  audit  process.  This  tool  can 
be  used  to  determine  the  extent  of  use  of  process-related  factors  when  defining  and 
measuring  success  with  relation  to  service  contracts.  If  the  correct  processes  are 
being  followed  in  a  proper  way,  then  it  is  only  natural  that  desired,  or  successful, 
outcomes  will  follow.  A  formal  audit  board  should  be  staffed  with  their  sole 
responsibility  consisting  of  conducting  audits  and  assessing  the  extent  of  proper  use 
of  processes  in  the  service  contracting  process. 

The  DoD  should  implement  the  use  of  program  management  concepts  to  the 
services  acquisition  process.  This  would  place  a  more  rigid  structure  on  how 
services  are  procured  and  the  contracts  subsequently  administered.  It  would  also 
ensure  the  involvement  of  PMs.  Of  the  three  major  stakeholders,  PMs  made  up  the 
smallest  portion  or  respondents.  Previous  research  also  notes  the  apparent  lack  of 
PMs  involved  in  the  process  (Miller  et  al.,  2012).  The  program  management 
approach  also  forces  the  PM  to  be  accountable  to  the  MDA,  who  is  briefed 
periodically  throughout  the  life  cycle.  Program  management  concepts  dictate  the 
utilization  of  IPTs  so  that  there  is  no  lack  of  knowledge  at  any  point  in  the  acquisition 
process.  The  DoD  would  benefit  from  the  application  of  program  management 
concepts  to  service  acquisition. 

A  final  recommendation  is  that  a  standardized  reporting  process  should  be  in 
place  in  order  to  track  contractor  performance  related  to  both  processes  and 
outcomes.  The  COR  should  be  intimately  involved  in  this  reporting  process  and 
status  needs  to  be  regularly  conveyed  to  the  stakeholders.  This  report  should 
include  cost  elements,  as  well  as  schedule  and  performance  elements.  However,  it 
need  not  be  limited  to  only  those  factors.  If  CORs  are  forced  to  report  on  adherence 
to  a  communication  plan  as  well  as  customer  satisfaction,  the  stakeholders  could 
also  track  how  effectively  processes  are  being  followed.  Typically,  customer 
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satisfaction  surveys  deal  with  outcomes,  but  they  could  also  refer  to 
communications,  planning,  and  administration.  If  a  standardized  reporting  process 
were  in  place,  it  might  be  possible  to  accurately  capture  metrics  in  order  to  define 
and  measure  the  success  of  a  service  contract. 

C.  Areas  for  Further  Research 

The  research  participants  included  in  this  study  were  strictly  from  United 
States  Navy  commands.  Due  to  this  single  Service  perspective,  we  recommend 
expanding  this  research  to  include  participants  from  the  United  States  Marine  Corps, 
Army,  and  Air  Force.  We  were  unable  to  incorporate  data  from  customers  and 
contractors  for  DoD  services  acquisitions.  As  such,  we  recommend  further  research 
include  a  more  complete  mix  of  stakeholders  by  including  contractors  and 
customers. 

NAVSEA  employs  a  “Services  Court”  process  to  strengthen  oversight  and 
improve  service  acquisition  outcomes.  These  newly  implemented  internal  control 
processes  provide  an  excellent  opportunity  for  future  research  directed  towards 
measuring  their  effectiveness  in  improving  services  acquisition  outcomes. 
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Appendix  A:  Tables  of  Survey  Results 


Table  A1.  Aggregate  Stakeholders’  Means  With  Distributions 


Aggregate 

Stakeholders 

Define 

Measure 

Likert 

Mean 

1 

2 

3 

4 

5 

Likert 

Mean 

1 

2 

3 

4 

5 

Process 

3.97 

31 

103 

147 

348 

419 

2.47 

85 

83 

57 

44 

26 

Outcome 

4.08 

9 

58 

161 

537 

437 

3.80 

46 

119 

219 

363 

198 

Cost 

4.03 

3 

19 

49 

158 

106 

4.04 

0 

3 

27 

90 

39 

Schedule 

3.93 

2 

29 

56 

164 

96 

4.00 

1 

5 

23 

96 

36 

Performance 

4.29 

4 

10 

56 

215 

235 

3.36 

45 

111 

169 

177 

123 

Table  A2.  Aggregate  Stakeholders’  Highest  Ranking  % 


Aggregate 

Stakeholders 

Define 

Measure 

Highest  Rank  % 

Highest  Rank  % 

Process 

40% 

46% 

Outcome 

60% 

54% 

Cost 

15% 

19% 

Schedule 

19% 

12% 

Performance 

26% 

24% 

Table  A3.  Stakeholder  Means  With  Distributions 


Stakeholders 

Define 

Measure 

Likert 

Mean 

1 

2 

3 

4 

5 

Likert 

Mean 

1 

2 

3 

4 

5 

PM 

Process 

3.87 

1 

11 

11 

30 

30 

2.52 

4 

8 

8 

4 

1 

Outcome 

4.13 

0 

4 

10 

40 

39 

3.78 

8 

8 

21 

25 

18 

Cost 

4.24 

0 

0 

3 

15 

9 

4.15 

0 

0 

1 

10 

3 

Schedule 

3.63 

0 

4 

6 

9 

7 

3.92 

0 

1 

2 

8 

3 
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Performance 

4.51 

0 

0 

1 

16 

23 

3.25 

8 

7 

18 

7 

12 

COR 

Process 

3.97 

8 

12 

19 

55 

62 

2.76 

6 

15 

7 

11 

3 

Outcome 

4.15 

0 

9 

15 

52 

68 

3.77 

10 

22 

39 

49 

23 

Cost 

4.16 

0 

3 

5 

23 

19 

4.23 

0 

0 

3 

11 

8 

Schedule 

4.06 

0 

4 

8 

21 

19 

4.08 

0 

1 

2 

16 

6 

Performance 

4.23 

0 

2 

2 

8 

30 

2.99 

10 

21 

34 

22 

9 

CO 

Process 

3.97 

22 

75 

112 

239 

307 

2.41 

71 

56 

40 

28 

21 

Outcome 

4.06 

7 

43 

122 

390 

305 

3.81 

28 

82 

149 

271 

153 

Cost 

3.98 

3 

16 

40 

106 

76 

3.98 

0 

3 

23 

63 

27 

Schedule 

3.91 

2 

20 

39 

127 

63 

3.98 

1 

3 

19 

67 

26 

Performance 

4.27 

2 

7 

43 

157 

166 

3.47 

27 

76 

107 

141 

100 
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Table  A4.  Stakeholder  Highest  Ranking  % 


Stakeholders 

Define 

Measure 

Highest  Rank  % 

Highest  Rank  % 

PM 

Process 

41% 

43% 

Outcome 

59% 

57% 

Cost 

11% 

13% 

Schedule 

22% 

14% 

Performance 

27% 

30% 

COR 

Process 

39% 

39% 

Outcome 

61% 

61% 

Cost 

17% 

28% 

Schedule 

19% 

9% 

Performance 

24% 

24% 

CO 

Process 

40% 

49% 

Outcome 

60% 

51% 

Cost 

15% 

17% 

Schedule 

17% 

12% 

Performance 

28% 

23% 
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Table  A5.  DAWIA  Level  Means  With  Distributions 


DAWIA 

LEVELS 

Define 

Measure 

Likert 

Mean 

1 

2 

3 

4 

5 

Likert 

Mean 

1 

2 

3 

4 

5 

LEVEL  1 

Process 

4.00 

9 

29 

47 

125 

131 

2.20 

23 

23 

23 

15 

8 

Outcome 

4.14 

1 

10 

56 

166 

161 

3.76 

16 

40 

77 

108 

65 

Cost 

4.12 

0 

2 

16 

51 

41 

3.88 

0 

0 

9 

24 

18 

Schedule 

3.96 

0 

6 

20 

53 

34 

3.96 

0 

2 

6 

33 

12 

Performance 

4.35 

1 

2 

20 

62 

86 

3.44 

16 

38 

62 

51 

35 

LEVEL  II 

Process 

4.16 

7 

19 

47 

103 

160 

2.63 

24 

22 

19 

16 

11 

Outcome 

4.17 

2 

9 

49 

174 

150 

3.86 

13 

30 

64 

120 

67 

Cost 

4.14 

2 

3 

14 

47 

42 

4.08 

0 

0 

6 

34 

10 

Schedule 

4.03 

0 

6 

17 

55 

33 

4.02 

0 

0 

9 

30 

10 

Performance 

4.35 

0 

0 

18 

72 

75 

3.49 

13 

30 

49 

56 

47 

LEVEL  III 

Process 

3.79 

15 

55 

53 

120 

128 

2.22 

38 

38 

15 

13 

7 

Outcome 

4.01 

6 

39 

56 

197 

126 

3.72 

17 

49 

78 

135 

66 

Cost 

3.76 

1 

14 

19 

60 

23 

3.88 

0 

3 

12 

32 

11 

Schedule 

4.01 

2 

17 

19 

56 

29 

3.95 

1 

3 

8 

33 

14 

Performance 

4.17 

3 

8 

18 

81 

74 

3.34 

16 

43 

58 

70 

41 
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Table  A6.  DAWIA  Level  Highest  Rank  % 


Stakeholders 

Define 

Measure 

Highest  Rank  % 

Highest  Rank  % 

Level  1 

Process 

42% 

49% 

Outcome 

58% 

61% 

Cost 

19% 

16% 

Schedule 

14% 

12% 

Performance 

25% 

23% 

Level  II 

Process 

39% 

48% 

Outcome 

61% 

52% 

Cost 

12% 

14% 

Schedule 

19% 

14% 

Performance 

30% 

23% 

Level  III 

Process 

40% 

43% 

Outcome 

60% 

57% 

Cost 

15% 

21% 

Schedule 

19% 

12% 

Performance 

27% 

25% 
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Table  A7.  Service  Type  Means  With  Distributions 


Service 

Type 

Define 

Measure 

Likert 

Mean 

1 

2 

3 

4 

5 

Likert 

Mean 

1 

2 

3 

4 

5 

Equipment  Related 

Process 

4.05 

7 

25 

69 

147 

159 

2.68 

27 

25 

23 

17 

14 

Outcome 

4.10 

2 

13 

70 

221 

160 

3.76 

17 

44 

94 

143 

72 

Cost 

4.03 

1 

5 

21 

64 

39 

4.08 

0 

0 

12 

39 

17 

Schedule 

3.97 

0 

6 

25 

70 

33 

3.87 

0 

2 

14 

39 

10 

Performance 

4.28 

1 

2 

24 

87 

88 

3.34 

17 

42 

68 

65 

45 

Knowledge  E 

ased 

Process 

3.92 

15 

59 

52 

138 

179 

2.29 

43 

37 

20 

19 

7 

Outcome 

4.02 

6 

38 

60 

223 

180 

3.82 

20 

46 

88 

155 

88 

Cost 

4.00 

1 

11 

19 

65 

45 

3.97 

0 

2 

12 

39 

14 

Schedule 

3.82 

2 

21 

20 

63 

41 

4.07 

1 

2 

8 

37 

20 

Performance 

4.24 

3 

6 

21 

95 

94 

3.41 

19 

42 

68 

79 

54 
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Table  A8.  Service  Type  Highest  Rank  % 


Service  Type 

Define 

Measure 

Highest  Rank  % 

Highest  Rank  % 

Equipment  Related 

Process 

42% 

44% 

Outcome 

58% 

56% 

Cost 

15% 

21% 

Schedule 

18% 

13% 

Performance 

25% 

22% 

Knowledge  Based 

Process 

39% 

48% 

Outcome 

61% 

52% 

Cost 

16% 

16% 

Schedule 

18% 

11% 

Performance 

28% 

25% 
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Appendix  B:  Statistical  Analysis  Results 


Table  B1.  Statistical  Analysis  of  Process  Definition  Between  CORs  and  COs 

PROCESS  DEFINE 


F-Test  Two-Sample  for  Variances 


COR 

CO 

Mean 

3.96795 

3.971164 

Variance 

0.26893 

0.3253929 

Observations 

26 

126 

df 

25 

125 

F 

0.82648 

P(F  <=  f)  one-tail 

0.29793 

F  Critical  one-tail 

0.56619 

t-Test:  Two-Sample  Assuming  Equal  Variances 

COR 

CO 

Mean 

3.96795 

3.971164 

Variance 

0.26893 

0.3253929 

Observations 

26 

126 

Pooled  Variance 

0.31598 

Hypothesized  Mean 
Difference 

0 

df 

150 

tStat 

-0.0266 

P(T  <=  t)  one-tail 

0.48943 

t  Critical  one-tail 

1 .65508 

P(T  <=  t)  two-tail 

0.97885 

t  Critical  two-tail 

1.97591 
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These  tables  display  the  statistical  analysis  conducted  to  determine  if  the 
mean  of  Likert  scale  responses  with  regards  to  the  importance  of  processes  when 
defining  success  from  the  COR  and  CO  showed  any  significant  difference.  The  F- 
test  shows  that  the  variance  between  responses  is  assumed  to  be  0.  The  t-test 
shows  that  there  is  no  significant  difference. 

Table  B2.  Statistical  Analysis  of  Outcome  Definition  Between  CORs  and  COs 

OUTCOME  DEFINE 


F-Test  Two-Sample  for  Variances 


COR 

CO 

Mean 

4.14744 

4.05776 

Variance 

0.25258 

0.27918 

Observations 

26 

126 

df 

25 

125 

F 

0.90471 

P(F  <=  f)  one-tail 

0.40164 

F  Critical  one-tail 

0.56619 

t-Test:  Two-Sample  Assuming  Equal  Variances 


COR 

CO 

Mean 

4.14744 

4.05776 

Variance 

0.25258 

0.27918 

Observations 

26 

126 

Pooled  Variance 

0.27475 

Hypothesized  Mean 
Difference 

0 

df 

150 

t  Stat 

0.79425 

P(T  <=  t)  one-tail 

0.21415 

t  Critical  one-tail 

1 .65508 
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P(T  <=  t)  two-tail 
t  Critical  two-tail 


0.4283 

1.97591 


These  tables  display  the  statistical  analysis  conducted  to  determine  if  the 
mean  of  Likert  scale  responses  with  regards  to  the  importance  of  outcomes  when 
defining  success  from  the  COR  and  CO  showed  any  significant  difference.  The  F- 
test  shows  that  the  variance  between  responses  is  assumed  to  be  0.  The  t-test 
shows  that  there  is  no  significant  difference. 

Table  B3.  Statistical  Analysis  of  Process  Measurement  Between  CORs  and 

COs 

PROCESS  MEASURE 


F-Test  Two-Sample  for  Variances 


COR 

CO 

Mean 

2.79167 

2.40789 

Variance 

1.10688 

0.99365 

Observations 

24 

114 

df 

23 

113 

F 

1.11395 

P(F  <=  f)  one-tail 

0.34183 

F  Critical  one-tail 

1 .62527 

t-Test:  Two-Sample  Assuming  Equal  Variances 


COR 

CO 

Mean 

2.79167 

2.40789 

Variance 

1.10688 

0.99365 

Observations 

24 

114 

Pooled  Variance 

1.0128 

Hypothesized  Mean 
Difference 

0 

df 

136 
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t  Stat 

1.69797 

P(T  <=  t)  one-tail 

0.0459 

t  Critical  one-tail 

1.65613 

P(T  <=  t)  two-tail 

0.0918 

t  Critical  two-tail 

1 .97756 

These  tables  display  the  statistical  analysis  conducted  to  determine  if  the 
mean  of  Likert  scale  responses  with  regards  to  the  importance  of  processes  when 
measuring  success  from  the  COR  and  CO  showed  any  significant  difference.  The 
F-test  shows  that  the  variance  between  responses  is  assumed  to  be  0.  The  t-test 
shows  that  there  is  a  statistically  significant  difference. 

Table  B4.  Statistical  Analysis  of  Outcome  Measurement  Between  CORs  and 

COs 

OUTCOME  MEASURE 


F-Test  Two-Sample  for  Variances 


COR 

CO 

Mean 

3.49028 

3.47899 

Variance 

0.28015 

0.33389 

Observations 

25 

118 

df 

24 

117 

F 

0.83905 

P(F  <=  f)  one-tail 

0.31894 

F  Critical  one-tail 

0.55834 

t-Test:  Two-Sample  Assuming  Equal  Variances 

COR 

CO 

Mean 

3.49028 

3.47899 

Variance 

0.28015 

0.33389 

Observations 

25 

118 

Pooled  Variance 

0.32475 
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Hypothesized  Mean 
Difference 

0 

df 

141 

t  Stat 

0.08997 

P(T  <=  t)  one-tail 

0.46422 

t  Critical  one-tail 

1 .65573 

P(T  <=  t)  two-tail 

0.92844 

t  Critical  two-tail 

1.97693 

These  tables  display  the  statistical  analysis  conducted  to  determine  if  the 
mean  of  Likert  scale  responses  with  regards  to  the  importance  of  outcomes  when 
measuring  success  from  the  COR  and  CO  showed  any  significant  difference.  The 
F-test  shows  that  the  variance  between  responses  is  assumed  to  be  0.  The  t-test 
shows  that  there  is  no  significant  difference. 

Table  B5.  Statistical  Analysis  of  Process  Definition  Between  CORs  and  PMs 

PROCESS  DEFINE 


F-Test  Two-Sample  for  Variances 


COR 

PM 

Mean 

3.96795 

3.86667 

Variance 

0.26893 

0.20704 

Observations 

26 

13 

df 

25 

12 

F 

1.29895 

P(F  <=  f)  one-tail 

0.32559 

F  Critical  one-tail 

2.49773 

t-Test:  Two-Sample  Assuming  Equal  Variances 

COR 

PM 

Mean 

3.96795 

3.86667 

Variance 

0.26893 

0.20704 
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Observations 
Pooled  Variance 


26 

0.24886 


13 


Hypothesized  Mean 
Difference 

0 

df 

37 

tStat 

0.5977 

P(T  <=  t)  one-tail 

0.27684 

t  Critical  one-tail 

1.68709 

P(T  <=  t)  two-tail 

0.55368 

t  Critical  two-tail 

2.02619 

These  tables  display  the  statistical  analysis  conducted  to  determine  if  the 
mean  of  Likert  scale  responses  with  regards  to  the  importance  of  processes  when 
defining  success  from  the  COR  and  PM  showed  any  significant  difference.  The  F- 
test  shows  that  the  variance  between  responses  is  assumed  to  be  0.  The  t-test 
shows  that  there  is  no  significant  difference. 

Table  B6.  Statistical  Analysis  of  Outcome  Definition  Between  CORs  and  PMs 

OUTCOME  DEFINE 


F-Test  Two-Sample  for  Variances 


COR 

PM 

Mean 

4.14744 

4.11538 

Variance 

0.25258 

0.17359 

Observations 

26 

13 

df 

25 

12 

F 

1 .45502 

P(F  <=  f)  one-tail 

0.25204 

F  Critical  one-tail 

2.49773 

t-Test:  Two-Sample  Assuming  Equal  Variances 

COR  PM~ 
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Mean 

4.14744 

4.11538 

Variance 

0.25258 

0.17359 

Observations 

26 

13 

Pooled  Variance 

0.22696 

Hypothesized  Mean 
Difference 

0 

df 

37 

t  Stat 

0.19806 

P(T  <=  t)  one-tail 

0.42204 

t  Critical  one-tail 

1.68709 

P(T  <=  t)  two-tail 

0.84408 

t  Critical  two-tail 

2.02619 

These  tables  display  the  statistical  analysis  conducted  to  determine  if  the 
mean  of  Likert  scale  responses  with  regards  to  the  importance  of  outcomes  when 
defining  success  from  the  COR  and  PM  showed  any  significant  difference.  The  F- 
test  shows  that  the  variance  between  responses  is  assumed  to  be  0.  The  t-test 
shows  that  there  is  no  significant  difference. 

Table  B7.  Statistical  Analysis  of  Process  Measurement  Between  CORs  and 

PMs 

PROCESS  MEASURE 


F-Test  Two-Sample  for  Variances 


COR 

PM 

Mean 

2.88248 

2.46154 

Variance 

1.13514 

0.8109 

Observations 

26 

13 

df 

25 

12 

F 

1.39986 

P(F  <=  f)  one-tail 

0.27594 

F  Critical  one-tail 

2.49773 
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t-Test:  Two-Sample  Assuming  Equal  Variances 


COR 

PM 

Mean 

2.88248 

2.46154 

Variance 

1.13514 

0.8109 

Observations 

26 

13 

Pooled  Variance 

1.02998 

Hypothesized  Mean 
Difference 

0 

df 

37 

tStat 

1.22104 

P(T  <=  t)  one-tail 

0.1149 

t  Critical  one-tail 

1.68709 

P(T  <=  t)  two-tail 

0.22979 

t  Critical  two-tail 

2.02619 

These  tables  display  the  statistical  analysis  conducted  to  determine  if  the 
mean  of  Likert  scale  responses  with  regards  to  the  importance  of  processes  when 
measuring  success  from  the  COR  and  PM  showed  any  significant  difference.  The 
F-test  shows  that  the  variance  between  responses  is  assumed  to  be  0.  The  t-test 
shows  that  there  is  no  significant  difference. 

Table  B8.  Statistical  Analysis  of  Outcome  Measurement  Between  CORs  and 

PMs 

OUTCOME  MEASURE 


F-Test  Two-Sample  for  Variances 


COR 

PM 

Mean 

3.49028 

3.44231 

Variance 

0.28015 

0.26853 

Observations 

25 

13 

df 

24 

12 
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F 

P(F  <=  f)  one-tail 
F  Critical  one-tail 


1 .04328 
0.48911 
2.50548 


t-Test:  Two-Sample  Assuming  Equal  Variances 


COR 

PM 

Mean 

3.49028 

3.44231 

Variance 

0.28015 

0.26853 

Observations 

25 

13 

Pooled  Variance 

0.27628 

Hypothesized  Mean 
Difference 

0 

df 

36 

t  Stat 

0.2669 

P(T  <=  t)  one-tail 

0.39553 

t  Critical  one-tail 

1.6883 

P(T  <=  t)  two-tail 

0.79107 

t  Critical  two-tail 

2.02809 

These  tables  display  the  statistical  analysis  conducted  to  determine  if  the 
mean  of  Likert  scale  responses  with  regards  to  the  importance  of  outcomes  when 
measuring  success  from  the  COR  and  PM  showed  any  significant  difference.  The 
F-test  shows  that  the  variance  between  responses  is  assumed  to  be  0.  The  t-test 
shows  that  there  is  no  significant  difference. 
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Table  B9.  Statistical  Analysis  of  Process  Definition  Between  COs  and  PMs 

PROCESS  DEFINE 


F-Test  Two-Sample  for  Variances 


CO 

PM 

Mean 

3.97116 

3.86667 

Variance 

0.32539 

0.20704 

Observations 

126 

13 

df 

125 

12 

F 

1.57167 

P(F  <=  f)  one-tail 

0.19214 

F  Critical  one-tail 

2.33924 

t-Test:  Two-Sample  Assuming  Equal  Variances 


CO 

PM 

Mean 

3.97116 

3.86667 

Variance 

0.32539 

0.20704 

Observations 

126 

13 

Pooled  Variance 

0.31503 

Hypothesized  Mean 
Difference 

0 

df 

137 

tStat 

0.63912 

P(T  <=  t)  one-tail 

0.26191 

t  Critical  one-tail 

1 .65605 

P(T  <=  t)  two-tail 

0.52381 

t  Critical  two-tail 

1 .97743 

These  tables  display  the  statistical  analysis  conducted  to  determine  if  the 
mean  of  Likert  scale  responses  with  regards  to  the  importance  of  processes  when 
defining  success  from  the  CO  and  PM  showed  any  significant  difference.  The  F-test 
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shows  that  the  variance  between  responses  is  assumed  to  be  0.  The  t-test  shows 
that  there  is  no  significant  difference. 

Table  BIO.  Statistical  Analysis  of  Outcome  Definition  Between  COs  and  PMs 

OUTCOME  DEFINE 


F-Test  Two-Sample  for  Variances 


CO 

PM 

Mean 

4.05776 

4.11538 

Variance 

0.27918 

0.17359 

Observations 

126 

13 

df 

125 

12 

F 

1.60826 

P(F  <=  f)  one-tail 

0.17953 

F  Critical  one-tail 

2.33924 

t-Test:  Two-Sample  Assuming  Equal  Variances 


CO 

PM 

Mean 

4.05776 

4.11538 

Variance 

0.27918 

0.17359 

Observations 

126 

13 

Pooled  Variance 

0.26993 

Hypothesized  Mean 
Difference 

0 

df 

137 

tStat 

-0.3807 

P(T  <=  t)  one-tail 

0.35199 

t  Critical  one-tail 

1 .65605 

P(T  <=  t)  two-tail 

0.70399 

t  Critical  two-tail 

1 .97743 
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These  tables  display  the  statistical  analysis  conducted  to  determine  if  the 
mean  of  Likert  scale  responses  with  regards  to  the  importance  of  outcomes  when 
defining  success  from  the  CO  and  PM  showed  any  significant  difference.  The  F-test 
shows  that  the  variance  between  responses  is  assumed  to  be  0.  The  t-test  shows 
that  there  is  no  significant  difference. 

Table  B11.  Statistical  Analysis  of  Process  Measurement  Between  COs  and 

PMs 

PROCESS  MEASURE 


F-Test  Two-Sample  for  Variances 


CO 

PM 

Mean 

2.40789 

2.46154 

Variance 

0.99365 

0.8109 

Observations 

114 

13 

df 

113 

12 

F 

1 .22538 

P(F  <=  f)  one-tail 

0.36651 

F  Critical  one-tail 

2.34371 

t-Test:  Two-Sample  Assuming  Equal  Variances 


CO 

PM 

Mean 

2.40789 

2.46154 

Variance 

0.99365 

0.8109 

Observations 

114 

13 

Pooled  Variance 

0.97611 

Hypothesized  Mean 
Difference 

0 

df 

125 

tStat 

-0.1855 

P(T  <=  t)  one-tail 

0.42658 

t  Critical  one-tail 

1.65714 
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P(T  <=  t)  two-tail 
t  Critical  two-tail 


0.85315 

1.97912 


These  tables  display  the  statistical  analysis  conducted  to  determine  if  the 
mean  of  Likert  scale  responses  with  regards  to  the  importance  of  processes  when 
measuring  success  from  the  CO  and  PM  showed  any  significant  difference.  The  F- 
test  shows  that  the  variance  between  responses  is  assumed  to  be  0.  The  t-test 
shows  that  there  is  no  significant  difference. 

Table  B12.  Statistical  Analysis  of  Outcome  Measurement  Between  COs  and 

PMs 

OUTCOME  MEASURE 


F-Test  Two-Sample  for  Variances 


CO 

PM 

Mean 

3.47899 

3.44231 

Variance 

0.33389 

0.26853 

Observations 

118 

13 

df 

117 

12 

F 

1.24341 

P(F  <=  f)  one-tail 

0.35453 

F  Critical  one-tail 

2.34212 

t-Test:  Two-Sample  Assuming  Equal  Variances 

CO 

PM 

Mean 

3.47899 

3.44231 

Variance 

0.33389 

0.26853 

Observations 

118 

13 

Pooled  Variance 

0.32781 

Hypothesized  Mean 
Difference 

0 

df  129 
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t  Stat 


0.21924 
0.4134 
1 .65675 
0.82681 
1.97852 


P(T  <=  t)  one-tail 
t  Critical  one-tail 
P(T  <=  t)  two-tail 
t  Critical  two-tail 


These  tables  display  the  statistical  analysis  conducted  to  determine  if  the 
mean  of  Likert  scale  responses  with  regards  to  the  importance  of  outcomes  when 
measuring  success  from  the  CO  and  PM  showed  any  significant  difference.  The  F- 
test  shows  that  the  variance  between  responses  is  assumed  to  be  0.  The  t-test 
shows  that  there  is  no  significant  difference. 
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